Jul 02 23:08:45.384683 ip-10-0-133-16 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 23:08:45.384699 ip-10-0-133-16 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 23:08:45.384708 ip-10-0-133-16 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 23:08:45.385090 ip-10-0-133-16 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 23:08:55.511236 ip-10-0-133-16 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 23:08:55.511255 ip-10-0-133-16 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 881d4af3380b410ab18206169c2d2c52 -- Jul 02 23:10:52.039658 ip-10-0-133-16 systemd[1]: Starting Kubernetes Kubelet... Jul 02 23:10:52.449288 ip-10-0-133-16 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:10:52.449288 ip-10-0-133-16 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 23:10:52.449288 ip-10-0-133-16 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:10:52.449288 ip-10-0-133-16 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 23:10:52.449288 ip-10-0-133-16 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.451241 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455637 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455656 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455663 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455668 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455672 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455675 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455679 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455683 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455688 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455692 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455695 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455699 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455703 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455709 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455714 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455720 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:52.588423 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455724 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:52.523931 ip-10-0-133-16 systemd[1]: Started Kubernetes Kubelet. Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455730 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455734 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455738 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455742 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455746 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455749 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455754 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455757 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455761 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455765 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455770 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455773 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455778 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455782 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455787 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455791 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455794 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455799 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455803 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:52.593736 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455807 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455810 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455814 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455818 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455822 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455825 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455828 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455832 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455836 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455839 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455843 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455846 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455850 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455853 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455856 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455860 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455864 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455868 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455872 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455876 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:52.594415 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455880 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455884 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455887 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455891 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455894 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455899 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455904 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455908 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455912 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455916 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455921 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455925 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455928 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455932 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455936 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455939 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455942 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455946 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455950 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455954 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:52.595170 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455957 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455961 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455965 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455968 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455972 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455976 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455979 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455983 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455987 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455991 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455995 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.455999 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456003 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456007 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456010 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456014 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456017 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456023 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456026 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.074032 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456030 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456034 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456037 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.456041 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457047 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457062 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457067 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457071 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457075 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457079 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457083 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457087 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457091 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457094 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457098 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457102 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457106 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457110 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457114 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457117 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:54.090208 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457121 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457125 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457129 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457132 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457136 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457140 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457144 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457148 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457151 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457155 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457159 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457163 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457167 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457171 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457174 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457178 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457182 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457186 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457190 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457194 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:54.090975 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457198 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457201 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457205 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457209 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457212 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457216 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457220 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457224 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457227 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457231 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457235 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457240 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457244 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457248 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457251 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457255 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457259 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457263 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457267 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457270 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:54.091705 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457273 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457278 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457282 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457286 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457290 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457309 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457313 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457317 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457323 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457328 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457332 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457336 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457340 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457344 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457348 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457351 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457355 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457358 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457363 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457366 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:54.092371 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457370 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457373 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457380 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457384 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457388 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457392 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457396 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457400 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457404 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457407 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457411 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457415 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457418 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457424 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457429 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457433 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457438 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457442 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457446 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457450 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.093158 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457455 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457461 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.457465 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458053 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458060 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458064 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458069 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458073 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458077 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458081 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458085 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458090 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458094 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458097 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458101 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458105 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458109 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458113 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458116 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458121 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:54.093789 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458124 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458129 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458132 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458136 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458139 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458143 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458147 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458150 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458156 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458161 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458166 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458170 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458174 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458180 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458184 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458188 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458193 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458196 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458202 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:54.094399 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458206 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458210 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458214 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458218 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458221 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458225 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458228 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458233 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458236 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458240 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458244 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458247 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458254 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458258 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458262 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458265 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458269 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458273 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458277 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458281 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.094939 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458285 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458305 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458310 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458313 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458317 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458321 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458324 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458328 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458331 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458335 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458345 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458349 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458352 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458356 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458360 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458363 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458367 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458370 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458374 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458377 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.095429 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458381 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458384 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458388 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458392 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458397 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458401 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458404 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458410 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458414 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458418 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458423 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458427 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458431 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458434 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458438 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458442 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458445 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458449 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458453 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:54.095926 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458456 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458460 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458464 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.458468 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458559 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458569 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458578 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458584 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458590 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458595 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458602 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458608 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458613 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458617 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458622 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458626 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458631 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458638 2561 flags.go:64] FLAG: --cgroup-root="" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458642 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458646 2561 flags.go:64] FLAG: --client-ca-file="" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458650 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458654 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458661 2561 flags.go:64] FLAG: --cluster-domain="" Jul 02 23:10:54.096393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458669 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458674 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458679 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458684 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458695 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458699 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458703 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458709 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458713 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458717 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458721 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458726 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458730 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458736 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458742 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458747 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458751 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458755 2561 flags.go:64] FLAG: --enable-server="true" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458759 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458765 2561 flags.go:64] FLAG: --event-burst="100" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458769 2561 flags.go:64] FLAG: --event-qps="50" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458774 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458778 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458783 2561 flags.go:64] FLAG: --eviction-hard="" Jul 02 23:10:54.096962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458788 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458792 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458800 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458804 2561 flags.go:64] FLAG: --eviction-soft="" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458809 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458814 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458818 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458822 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458827 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458832 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458836 2561 flags.go:64] FLAG: --feature-gates="" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458842 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458846 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458850 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458855 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458859 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458863 2561 flags.go:64] FLAG: --help="false" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458867 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458872 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458877 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458881 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458886 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458890 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458895 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 23:10:54.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458899 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458904 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458909 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458913 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458917 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458922 2561 flags.go:64] FLAG: --kube-reserved="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458926 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458930 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458935 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458939 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458944 2561 flags.go:64] FLAG: --lock-file="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458948 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458952 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458957 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458964 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458968 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458974 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458978 2561 flags.go:64] FLAG: --logging-format="text" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458982 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458987 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458991 2561 flags.go:64] FLAG: --manifest-url="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.458996 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459002 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459006 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459013 2561 flags.go:64] FLAG: --max-pods="110" Jul 02 23:10:54.098186 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459017 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459021 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459025 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459030 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459034 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459039 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459043 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459054 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459058 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459063 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459067 2561 flags.go:64] FLAG: --pod-cidr="" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459071 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459079 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459083 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459088 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459092 2561 flags.go:64] FLAG: --port="10250" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459097 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459101 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0304b83096cb4c4eb" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459108 2561 flags.go:64] FLAG: --qos-reserved="" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459112 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459116 2561 flags.go:64] FLAG: --register-node="true" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459121 2561 flags.go:64] FLAG: --register-with-taints="" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459126 2561 flags.go:64] FLAG: --registry-burst="10" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459132 2561 flags.go:64] FLAG: --registry-qps="5" Jul 02 23:10:54.098809 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459137 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459141 2561 flags.go:64] FLAG: --reserved-memory="" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459146 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459151 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459155 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459160 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459164 2561 flags.go:64] FLAG: --runonce="false" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459168 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459172 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459176 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459181 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459185 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459190 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459194 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459199 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459203 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459208 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459212 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459216 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459221 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459225 2561 flags.go:64] FLAG: --system-cgroups="" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459229 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459237 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459240 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459245 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 23:10:54.099395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459250 2561 flags.go:64] FLAG: --tls-min-version="" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459256 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459260 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459264 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459268 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459272 2561 flags.go:64] FLAG: --v="2" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459279 2561 flags.go:64] FLAG: --version="false" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459285 2561 flags.go:64] FLAG: --vmodule="" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459306 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.459312 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459470 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459477 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459483 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459489 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459494 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459498 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459502 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459507 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459511 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459515 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459519 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:54.100021 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459524 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459528 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459532 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459537 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459541 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459545 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459549 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459553 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459556 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459559 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459563 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459567 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459572 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459575 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459579 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459583 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459587 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459592 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459596 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:54.100553 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459600 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459604 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459608 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459612 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459615 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459619 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459623 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459626 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459630 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459634 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459637 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459641 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459645 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459649 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459652 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459656 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459660 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459664 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459668 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459672 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:54.101040 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459676 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459679 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459683 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459687 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459691 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459697 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459701 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459704 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459709 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459712 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459718 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459722 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459726 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459730 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459733 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459739 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459743 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459749 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459752 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459756 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:54.101580 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459759 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459763 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459766 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459771 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459774 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459778 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459782 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459786 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459790 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459794 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459798 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459801 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459805 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459809 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459812 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459816 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459820 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459824 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459829 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459832 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:54.102066 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459836 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459840 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459845 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459849 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459853 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459856 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459860 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459863 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.459867 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.460571 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.469458 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.469480 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.469501 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.469510 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 23:10:54.102586 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469570 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469575 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469578 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469581 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469584 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469587 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469590 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469593 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469596 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469598 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469601 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469604 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469607 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469610 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469613 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469616 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469619 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469622 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469625 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469628 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:54.102957 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469631 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469634 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469637 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469640 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469643 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469645 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469649 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469651 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469654 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469657 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469659 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469662 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469665 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469667 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469670 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469673 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469675 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469678 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469680 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:54.103447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469683 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469685 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469688 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469691 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469693 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469696 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469699 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469705 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469708 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469711 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469713 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469716 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469720 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469724 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469727 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469730 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469733 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469736 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469739 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469742 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:54.103938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469746 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469748 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469751 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469754 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469756 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469759 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469762 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469764 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469767 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469769 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469772 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469774 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469777 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469780 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469782 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469785 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469788 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469790 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469793 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469798 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:54.104424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469801 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469804 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469807 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469809 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469812 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469816 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469820 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469823 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469827 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469830 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469833 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469836 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469839 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469841 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469844 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469846 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469849 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469851 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469854 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:54.104935 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469856 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.469862 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469978 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469983 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469986 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469989 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469992 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469995 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.469998 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470001 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470004 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470006 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470009 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470013 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 23:10:54.105414 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470017 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470021 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470025 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470028 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470031 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470034 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470037 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470040 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470043 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470045 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470048 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470051 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470053 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470056 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470059 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470061 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470064 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470066 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470069 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 23:10:54.105779 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470071 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470075 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470078 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470081 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470083 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470086 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470088 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470091 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470093 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470096 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470099 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470101 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470108 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470110 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470113 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470116 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470118 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470121 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470123 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470126 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 23:10:54.106235 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470128 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470131 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470134 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470136 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470139 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470141 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470144 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470146 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470149 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470151 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470154 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470156 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470159 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470161 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470164 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470167 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470169 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470171 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470174 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470176 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 23:10:54.106867 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470179 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470182 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470185 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470189 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470194 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470197 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470200 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470203 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470206 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470208 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470212 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470215 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470217 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470220 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470222 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470225 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470228 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470230 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 23:10:54.107385 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470233 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470235 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470238 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470240 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470243 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470246 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470248 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470251 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470253 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:52.470256 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.470261 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.470403 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.474477 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.475791 2561 server.go:1019] "Starting client certificate rotation" Jul 02 23:10:54.107840 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.475894 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.475932 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.478257 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.483011 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.491177 2561 log.go:25] "Validated CRI v1 runtime API" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.493253 2561 log.go:25] "Validated CRI v1 image API" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.493957 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.496756 2561 fs.go:135] Filesystem UUIDs: map[365fdaa6-470b-4cb1-85b1-ec338d283438:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 da35d00c-1f6e-4a4e-9dd3-e1b85e78f98d:/dev/nvme0n1p4] Jul 02 23:10:54.108182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.496772 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.501756 2561 manager.go:216] Machine: {Timestamp:2026-07-02 23:10:52.500448858 +0000 UTC m=+0.336879124 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101259 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e196b5ce5fafb0f51e23d1bcf3f1 SystemUUID:ec28e196-b5ce-5faf-b0f5-1e23d1bcf3f1 BootID:881d4af3-380b-410a-b182-06169c2d2c52 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:42:e2:43:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:42:e2:43:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:9d:49:b3:62:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.501852 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.501932 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.504975 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.505004 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.505725 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.505736 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.505743 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.506569 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.506840 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.507873 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510741 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510852 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510878 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510887 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510909 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 23:10:54.108388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510919 2561 kubelet.go:398] "Adding apiserver pod source" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.510927 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.511282 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.511318 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.513986 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.516083 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.516111 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517400 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517415 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517421 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517427 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517436 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517442 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517451 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517456 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517464 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517470 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517491 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.517497 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.518350 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.518357 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.520591 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.520625 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.520792 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.523145 2561 server.go:1268] "Started kubelet" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.523368 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.523549 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.523600 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.525277 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.525903 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.528395 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.529131 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-16.ec2.internal.18be9bd78a22ef2a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-16.ec2.internal,UID:ip-10-0-133-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-16.ec2.internal,},FirstTimestamp:2026-07-02 23:10:52.523114282 +0000 UTC m=+0.359544548,LastTimestamp:2026-07-02 23:10:52.523114282 +0000 UTC m=+0.359544548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-16.ec2.internal,}" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.531078 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.531578 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.531852 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532127 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532127 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532151 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532228 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532236 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532585 2561 factory.go:153] Registering CRI-O factory Jul 02 23:10:54.108856 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532600 2561 factory.go:223] Registration of the crio container factory successfully Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532665 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532679 2561 factory.go:55] Registering systemd factory Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532687 2561 factory.go:223] Registration of the systemd container factory successfully Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532710 2561 factory.go:103] Registering Raw factory Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.532723 2561 manager.go:1196] Started watching for new ooms in manager Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.533915 2561 manager.go:349] Starting recovery of all containers Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.542834 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zqcs" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.545835 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:52.545897 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.548552 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zqcs" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.549772 2561 manager.go:354] Recovery completed Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.553888 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.556409 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.556438 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.556451 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557027 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557041 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557063 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557042 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557778 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557805 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.557818 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.558327 2561 policy_none.go:49] "None policy: Start" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.558340 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.558350 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:52.559407 2561 policy_none.go:47] "Start" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.959394 2561 manager.go:337] "Starting Device Plugin manager" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:53.959643 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.959657 2561 server.go:87] "Starting device plugin registration server" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.960072 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.960084 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.960404 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.960460 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:53.960467 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:53.961030 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:53.961082 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.060362 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.060357 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061322 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061348 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061358 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061391 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061687 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061719 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.061736 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 23:10:54.109897 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.061769 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 23:10:54.111743 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.063275 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:54.111743 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.069622 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.111743 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.069643 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-16.ec2.internal\": node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.126123 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.126099 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.162233 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.162199 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal"] Jul 02 23:10:54.162329 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.162292 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.163566 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.163549 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.163667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.163580 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.163667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.163593 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.164913 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.164895 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.165038 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.165097 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165053 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.165693 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165672 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.165749 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165697 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.165749 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165710 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.165749 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165732 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.165841 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165751 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.165841 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.165761 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.166889 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.166876 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.166932 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.166901 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 23:10:54.167966 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.167952 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientMemory" Jul 02 23:10:54.168032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.167980 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 23:10:54.168032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.167994 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeHasSufficientPID" Jul 02 23:10:54.178217 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.178193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.178307 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.178223 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.178307 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.178243 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b956c94c524f1281d984172ee6c41e32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-16.ec2.internal\" (UID: \"b956c94c524f1281d984172ee6c41e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.198499 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.198472 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-16.ec2.internal\" not found" node="ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.203060 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.203041 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-16.ec2.internal\" not found" node="ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.227053 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.227023 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.278447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.278623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.278623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278496 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.278623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278525 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ac7136b72eeb4c1bec7700ebb2a216-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal\" (UID: \"27ac7136b72eeb4c1bec7700ebb2a216\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.278623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b956c94c524f1281d984172ee6c41e32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-16.ec2.internal\" (UID: \"b956c94c524f1281d984172ee6c41e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.278786 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.278643 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b956c94c524f1281d984172ee6c41e32-config\") pod \"kube-apiserver-proxy-ip-10-0-133-16.ec2.internal\" (UID: \"b956c94c524f1281d984172ee6c41e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.327758 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.327681 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.428291 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.428260 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.501805 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.501767 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.506672 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.506657 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.528611 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.528581 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.629200 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.629109 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.705794 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.705768 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:54.729414 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.729389 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-16.ec2.internal\" not found" Jul 02 23:10:54.779757 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.779731 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:54.819699 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.819668 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 23:10:54.820159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.819825 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 23:10:54.820159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.819835 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 23:10:54.820159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.819835 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 23:10:54.855912 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.855888 2561 apiserver.go:52] "Watching apiserver" Jul 02 23:10:54.876116 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.876089 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.876263 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.876146 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 23:10:54.880204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.880153 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 23:10:54.880487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.880469 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-c2pwk","openshift-ovn-kubernetes/ovnkube-node-xhdvp","kube-system/konnectivity-agent-5qjns","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8","openshift-dns/node-resolver-vp285","openshift-image-registry/node-ca-nwnt8","openshift-multus/multus-twd6k","openshift-network-operator/iptables-alerter-lgxxv","kube-system/global-pull-secret-syncer-8tm84","openshift-cluster-node-tuning-operator/tuned-4br4g","openshift-multus/multus-additional-cni-plugins-pmmhc","openshift-multus/network-metrics-daemon-hf4vd"] Jul 02 23:10:54.882299 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.882282 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:54.882386 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.882362 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:10:54.883841 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.883821 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.885523 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.885500 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:54.886463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.886439 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.886620 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.886440 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 23:10:54.886620 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.886545 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xqlnh\"" Jul 02 23:10:54.886620 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.886521 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 23:10:54.887165 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.887150 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.888062 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.888046 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 23:10:54.888062 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.888056 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.888168 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.888067 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 23:10:54.888514 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.888501 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:54.889488 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889473 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rgs2g\"" Jul 02 23:10:54.889566 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889521 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 23:10:54.889566 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889528 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 23:10:54.889860 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889842 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:54.889963 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889876 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 23:10:54.889963 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.889923 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.890095 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.890078 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.890325 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.890309 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rzkxm\"" Jul 02 23:10:54.891143 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891035 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 23:10:54.891143 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891118 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" Jul 02 23:10:54.891294 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891226 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.891773 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891740 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwbg9\"" Jul 02 23:10:54.891861 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891767 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.891861 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.891804 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.892835 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.892817 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 23:10:54.892958 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.892896 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.892958 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.892917 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mj9d8\"" Jul 02 23:10:54.893083 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.893064 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:54.893503 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.893481 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.894133 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894109 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 23:10:54.894244 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894177 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 23:10:54.894493 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894477 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9f7ph\"" Jul 02 23:10:54.894690 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894635 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 23:05:53 +0000 UTC" deadline="2028-03-08 06:36:41.832410271 +0000 UTC" Jul 02 23:10:54.894765 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894690 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14743h25m46.93772388s" Jul 02 23:10:54.894931 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.894914 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.895029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.895012 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.895029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.895041 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 23:10:54.895702 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.895450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:54.895702 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.895556 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:10:54.896762 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.896741 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2756p\"" Jul 02 23:10:54.897376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.897087 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.897376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.897171 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 23:10:54.897376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.897314 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.897565 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.897435 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.899124 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.899106 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.900199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900181 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:10:54.900372 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900351 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 23:10:54.900464 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900402 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 23:10:54.900464 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900455 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9sl5w\"" Jul 02 23:10:54.900813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900797 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal"] Jul 02 23:10:54.900859 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900823 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal"] Jul 02 23:10:54.901193 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.900892 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:54.901193 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:54.900949 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:10:54.901946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.901929 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sjgtt\"" Jul 02 23:10:54.902067 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.902049 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 23:10:54.902160 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.902090 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 23:10:54.909715 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.909696 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbztp" Jul 02 23:10:54.918426 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.918408 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbztp" Jul 02 23:10:54.977161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.977138 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 23:10:54.981849 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-hosts-file\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:54.981936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981854 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-var-lib-kubelet\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.981936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981870 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-log-socket\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.981936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981886 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:54.981936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-modprobe-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.981936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-kubelet\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981954 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-k8s-cni-cncf-io\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.981969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-bin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982014 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-etc-kubernetes\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-lib-modules\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.982092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982090 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-host\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-systemd-units\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982138 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-netd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cnibin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982184 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53f9d8b5-07f2-4d5f-93ae-fe9821920914-iptables-alerter-script\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-kubelet-config\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-systemd\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.982260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-var-lib-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982278 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-bin\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982299 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-env-overrides\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982318 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/248d33c7-27bc-469d-8bbe-53669184b131-agent-certs\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-conf-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982357 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tkt\" (UniqueName: \"kubernetes.io/projected/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-kube-api-access-77tkt\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982378 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-device-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982394 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/248d33c7-27bc-469d-8bbe-53669184b131-konnectivity-ca\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8610b55d-ca08-4394-a36c-001cb102480b-host\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmpd\" (UniqueName: \"kubernetes.io/projected/53f9d8b5-07f2-4d5f-93ae-fe9821920914-kube-api-access-slmpd\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:54.982483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-socket-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982490 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfv7\" (UniqueName: \"kubernetes.io/projected/46bec173-516c-426e-8db3-dca18369d958-kube-api-access-rxfv7\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysconfig\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982555 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-os-release\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982596 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-system-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982627 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53f9d8b5-07f2-4d5f-93ae-fe9821920914-host-slash\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982646 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqqfh\" (UniqueName: \"kubernetes.io/projected/8610b55d-ca08-4394-a36c-001cb102480b-kube-api-access-pqqfh\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-slash\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982715 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982741 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982765 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-os-release\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982784 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cni-binary-copy\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982803 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-daemon-config\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982817 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66r4\" (UniqueName: \"kubernetes.io/projected/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-kube-api-access-d66r4\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:54.982839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-etc-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982855 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-multus\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982869 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-tmp-dir\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982881 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-dbus\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982907 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-kubernetes\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-sys\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.982981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cnibin\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983003 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eec0b321-ccf0-4210-8f24-0b1061c43743-ovn-node-metrics-cert\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983016 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-tmp\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983040 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqrk\" (UniqueName: \"kubernetes.io/projected/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-kube-api-access-jmqrk\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983071 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6xc\" (UniqueName: \"kubernetes.io/projected/eec0b321-ccf0-4210-8f24-0b1061c43743-kube-api-access-px6xc\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-hostroot\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:54.983370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983163 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-sys-fs\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983182 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-conf\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983201 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-registration-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983218 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8610b55d-ca08-4394-a36c-001cb102480b-serviceca\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983251 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-netns\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983281 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-system-cni-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983306 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-netns\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983370 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-run\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-socket-dir-parent\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983412 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxgp\" (UniqueName: \"kubernetes.io/projected/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-kube-api-access-dzxgp\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983433 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-multus-certs\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983458 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-systemd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-etc-selinux\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-etc-tuned\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.983851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983501 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvcg\" (UniqueName: \"kubernetes.io/projected/a09249fe-61b2-4479-bf24-fcee604ce390-kube-api-access-6mvcg\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:54.984312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-ovn\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.984312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983526 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-node-log\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.984312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983557 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-config\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.984312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983584 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-script-lib\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:54.984312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:54.983598 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-kubelet\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.017288 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.017254 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ac7136b72eeb4c1bec7700ebb2a216.slice/crio-a80a9567907ea21b4eee0412aacca2487ccbe6c14902a21d5323f909e7a37f11 WatchSource:0}: Error finding container a80a9567907ea21b4eee0412aacca2487ccbe6c14902a21d5323f909e7a37f11: Status 404 returned error can't find the container with id a80a9567907ea21b4eee0412aacca2487ccbe6c14902a21d5323f909e7a37f11 Jul 02 23:10:55.017666 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.017646 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb956c94c524f1281d984172ee6c41e32.slice/crio-6b12f4c99f3afb761854a0d90a12f107a1054b39d53662e7f2ae96897ada7715 WatchSource:0}: Error finding container 6b12f4c99f3afb761854a0d90a12f107a1054b39d53662e7f2ae96897ada7715: Status 404 returned error can't find the container with id 6b12f4c99f3afb761854a0d90a12f107a1054b39d53662e7f2ae96897ada7715 Jul 02 23:10:55.022037 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.022007 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:10:55.065168 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.065126 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" event={"ID":"27ac7136b72eeb4c1bec7700ebb2a216","Type":"ContainerStarted","Data":"a80a9567907ea21b4eee0412aacca2487ccbe6c14902a21d5323f909e7a37f11"} Jul 02 23:10:55.066734 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.066704 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" event={"ID":"b956c94c524f1281d984172ee6c41e32","Type":"ContainerStarted","Data":"6b12f4c99f3afb761854a0d90a12f107a1054b39d53662e7f2ae96897ada7715"} Jul 02 23:10:55.084815 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-daemon-config\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.084880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084821 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66r4\" (UniqueName: \"kubernetes.io/projected/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-kube-api-access-d66r4\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.084880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-etc-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.084880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-multus\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.084880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-tmp-dir\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084895 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-dbus\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-kubernetes\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-sys\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084931 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-etc-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084954 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-multus\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.084981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cnibin\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085001 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-kubernetes\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eec0b321-ccf0-4210-8f24-0b1061c43743-ovn-node-metrics-cert\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085023 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-sys\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085057 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-tmp\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085072 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cnibin\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085128 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqrk\" (UniqueName: \"kubernetes.io/projected/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-kube-api-access-jmqrk\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085150 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6xc\" (UniqueName: \"kubernetes.io/projected/eec0b321-ccf0-4210-8f24-0b1061c43743-kube-api-access-px6xc\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085173 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-hostroot\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-sys-fs\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-conf\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085285 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-registration-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085317 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8610b55d-ca08-4394-a36c-001cb102480b-serviceca\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085338 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-netns\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085348 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-system-cni-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085374 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-tmp-dir\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085422 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-hostroot\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085497 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-dbus\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085524 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-daemon-config\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.085660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085524 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085567 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-netns\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085626 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-registration-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085667 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-netns\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085673 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-run\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085707 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-system-cni-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085713 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085732 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085746 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-netns\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085749 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-conf\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085749 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-run\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-socket-dir-parent\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085808 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085811 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-sys-fs\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.086499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxgp\" (UniqueName: \"kubernetes.io/projected/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-kube-api-access-dzxgp\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-multus-certs\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085860 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-systemd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-socket-dir-parent\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-etc-selinux\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085902 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-multus-certs\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085910 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-etc-tuned\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvcg\" (UniqueName: \"kubernetes.io/projected/a09249fe-61b2-4479-bf24-fcee604ce390-kube-api-access-6mvcg\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085934 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-systemd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-ovn\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085978 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-etc-selinux\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.085980 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-ovn\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-node-log\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-config\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-script-lib\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086110 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-node-log\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086161 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8610b55d-ca08-4394-a36c-001cb102480b-serviceca\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086166 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-kubelet\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.087284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086196 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-kubelet\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086223 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-hosts-file\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086248 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-var-lib-kubelet\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086267 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-hosts-file\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-log-socket\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-modprobe-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086326 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-var-lib-kubelet\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086347 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-kubelet\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-k8s-cni-cncf-io\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-bin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.086411 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-etc-kubernetes\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-lib-modules\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086466 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-log-socket\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.086480 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:10:55.58646076 +0000 UTC m=+2.078652430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:55.088181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086522 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-var-lib-cni-bin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086574 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-kubelet\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086576 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-lib-modules\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086615 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-etc-kubernetes\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086653 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysctl-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086656 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-host-run-k8s-cni-cncf-io\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086687 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-script-lib\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-host\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086718 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-ovnkube-config\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-systemd-units\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086737 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-host\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-netd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086774 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cnibin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086775 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-systemd-units\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53f9d8b5-07f2-4d5f-93ae-fe9821920914-iptables-alerter-script\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086798 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-modprobe-d\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086823 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-netd\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-kubelet-config\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.089081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086855 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-systemd\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086871 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cnibin\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086912 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-systemd\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086921 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086954 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-var-lib-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086912 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a7cc66b4-608b-4631-827a-2c764e4e023a-kubelet-config\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.086979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-bin\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087013 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-cni-bin\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087015 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-env-overrides\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/248d33c7-27bc-469d-8bbe-53669184b131-agent-certs\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087016 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-var-lib-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-conf-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tkt\" (UniqueName: \"kubernetes.io/projected/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-kube-api-access-77tkt\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087143 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-multus-conf-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-device-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087371 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53f9d8b5-07f2-4d5f-93ae-fe9821920914-iptables-alerter-script\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087401 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eec0b321-ccf0-4210-8f24-0b1061c43743-env-overrides\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.089721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087431 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087461 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-device-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087504 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087504 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/248d33c7-27bc-469d-8bbe-53669184b131-konnectivity-ca\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087596 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8610b55d-ca08-4394-a36c-001cb102480b-host\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087689 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmpd\" (UniqueName: \"kubernetes.io/projected/53f9d8b5-07f2-4d5f-93ae-fe9821920914-kube-api-access-slmpd\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087697 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8610b55d-ca08-4394-a36c-001cb102480b-host\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087738 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-socket-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfv7\" (UniqueName: \"kubernetes.io/projected/46bec173-516c-426e-8db3-dca18369d958-kube-api-access-rxfv7\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087812 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysconfig\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-os-release\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087880 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46bec173-516c-426e-8db3-dca18369d958-socket-dir\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087883 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-system-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53f9d8b5-07f2-4d5f-93ae-fe9821920914-host-slash\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087946 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-system-cni-dir\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:55.090213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.087966 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a09249fe-61b2-4479-bf24-fcee604ce390-etc-sysconfig\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqqfh\" (UniqueName: \"kubernetes.io/projected/8610b55d-ca08-4394-a36c-001cb102480b-kube-api-access-pqqfh\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088012 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-os-release\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-slash\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088094 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088123 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-os-release\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cni-binary-copy\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088174 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53f9d8b5-07f2-4d5f-93ae-fe9821920914-host-slash\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088234 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-run-openvswitch\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.088259 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088266 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-run-ovn-kubernetes\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088294 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eec0b321-ccf0-4210-8f24-0b1061c43743-host-slash\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088261 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-os-release\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.088307 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:10:55.588293382 +0000 UTC m=+2.080485035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088758 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-cni-binary-copy\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088855 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/248d33c7-27bc-469d-8bbe-53669184b131-konnectivity-ca\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:55.090696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088911 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-tmp\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.091157 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.088965 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eec0b321-ccf0-4210-8f24-0b1061c43743-ovn-node-metrics-cert\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.091157 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.089180 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a09249fe-61b2-4479-bf24-fcee604ce390-etc-tuned\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.091157 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.089675 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/248d33c7-27bc-469d-8bbe-53669184b131-agent-certs\") pod \"konnectivity-agent-5qjns\" (UID: \"248d33c7-27bc-469d-8bbe-53669184b131\") " pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:55.095924 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.095907 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:10:55.095924 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.095926 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:10:55.096070 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.095937 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:55.096070 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.095992 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:10:55.59598138 +0000 UTC m=+2.088173033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:55.096070 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.095934 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66r4\" (UniqueName: \"kubernetes.io/projected/b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13-kube-api-access-d66r4\") pod \"node-resolver-vp285\" (UID: \"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13\") " pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.097550 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.097516 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxgp\" (UniqueName: \"kubernetes.io/projected/4241c9b5-8638-4e3f-a515-8a9862cdf8a2-kube-api-access-dzxgp\") pod \"multus-twd6k\" (UID: \"4241c9b5-8638-4e3f-a515-8a9862cdf8a2\") " pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.098342 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.098321 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqrk\" (UniqueName: \"kubernetes.io/projected/1d12fc74-35e0-44d9-96e7-e3c99eb8cc29-kube-api-access-jmqrk\") pod \"multus-additional-cni-plugins-pmmhc\" (UID: \"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29\") " pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.098442 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.098425 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6xc\" (UniqueName: \"kubernetes.io/projected/eec0b321-ccf0-4210-8f24-0b1061c43743-kube-api-access-px6xc\") pod \"ovnkube-node-xhdvp\" (UID: \"eec0b321-ccf0-4210-8f24-0b1061c43743\") " pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.098762 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.098747 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvcg\" (UniqueName: \"kubernetes.io/projected/a09249fe-61b2-4479-bf24-fcee604ce390-kube-api-access-6mvcg\") pod \"tuned-4br4g\" (UID: \"a09249fe-61b2-4479-bf24-fcee604ce390\") " pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.102939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.102923 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfv7\" (UniqueName: \"kubernetes.io/projected/46bec173-516c-426e-8db3-dca18369d958-kube-api-access-rxfv7\") pod \"aws-ebs-csi-driver-node-v57s8\" (UID: \"46bec173-516c-426e-8db3-dca18369d958\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.103917 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.103896 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqqfh\" (UniqueName: \"kubernetes.io/projected/8610b55d-ca08-4394-a36c-001cb102480b-kube-api-access-pqqfh\") pod \"node-ca-nwnt8\" (UID: \"8610b55d-ca08-4394-a36c-001cb102480b\") " pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.104246 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.104232 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmpd\" (UniqueName: \"kubernetes.io/projected/53f9d8b5-07f2-4d5f-93ae-fe9821920914-kube-api-access-slmpd\") pod \"iptables-alerter-lgxxv\" (UID: \"53f9d8b5-07f2-4d5f-93ae-fe9821920914\") " pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.104792 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.104776 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tkt\" (UniqueName: \"kubernetes.io/projected/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-kube-api-access-77tkt\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:55.160669 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.160584 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:55.206548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.206515 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:10:55.209493 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.209469 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:10:55.216254 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.216225 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec0b321_ccf0_4210_8f24_0b1061c43743.slice/crio-d891c81342273346c0aa567a613298deaee4532f56cb2a46e2a0c45063cdb809 WatchSource:0}: Error finding container d891c81342273346c0aa567a613298deaee4532f56cb2a46e2a0c45063cdb809: Status 404 returned error can't find the container with id d891c81342273346c0aa567a613298deaee4532f56cb2a46e2a0c45063cdb809 Jul 02 23:10:55.218029 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.218003 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248d33c7_27bc_469d_8bbe_53669184b131.slice/crio-ee8b32f5ba942d1812ea780c4e15d87a7a1b7b37f66a7b440cb4192d284344bc WatchSource:0}: Error finding container ee8b32f5ba942d1812ea780c4e15d87a7a1b7b37f66a7b440cb4192d284344bc: Status 404 returned error can't find the container with id ee8b32f5ba942d1812ea780c4e15d87a7a1b7b37f66a7b440cb4192d284344bc Jul 02 23:10:55.222973 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.222956 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" Jul 02 23:10:55.231395 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.231371 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bec173_516c_426e_8db3_dca18369d958.slice/crio-8d0a9fba938d91b385c890a143f70f25e859e9a8352c9d5e2b30bd7b4a2253b7 WatchSource:0}: Error finding container 8d0a9fba938d91b385c890a143f70f25e859e9a8352c9d5e2b30bd7b4a2253b7: Status 404 returned error can't find the container with id 8d0a9fba938d91b385c890a143f70f25e859e9a8352c9d5e2b30bd7b4a2253b7 Jul 02 23:10:55.238276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.238258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vp285" Jul 02 23:10:55.244311 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.244292 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cbbc63_2b9c_4acf_9b24_1b8fe53b2e13.slice/crio-eb1f9869c3f2a90b95201aa697ab05f380e9e1f4fa167d943347cc989b714383 WatchSource:0}: Error finding container eb1f9869c3f2a90b95201aa697ab05f380e9e1f4fa167d943347cc989b714383: Status 404 returned error can't find the container with id eb1f9869c3f2a90b95201aa697ab05f380e9e1f4fa167d943347cc989b714383 Jul 02 23:10:55.256590 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.256569 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwnt8" Jul 02 23:10:55.273505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.273484 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-twd6k" Jul 02 23:10:55.280556 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.280519 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4241c9b5_8638_4e3f_a515_8a9862cdf8a2.slice/crio-aacb458465255890c02c2cd30f3150fdb8b96f913c61e3d4d15eebac95619434 WatchSource:0}: Error finding container aacb458465255890c02c2cd30f3150fdb8b96f913c61e3d4d15eebac95619434: Status 404 returned error can't find the container with id aacb458465255890c02c2cd30f3150fdb8b96f913c61e3d4d15eebac95619434 Jul 02 23:10:55.289575 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.289526 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lgxxv" Jul 02 23:10:55.295572 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.295548 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4br4g" Jul 02 23:10:55.297969 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.297941 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f9d8b5_07f2_4d5f_93ae_fe9821920914.slice/crio-762e0f94355583527393081eb8f829df7f14cf6eb85f7cd0a952a70e351bfc85 WatchSource:0}: Error finding container 762e0f94355583527393081eb8f829df7f14cf6eb85f7cd0a952a70e351bfc85: Status 404 returned error can't find the container with id 762e0f94355583527393081eb8f829df7f14cf6eb85f7cd0a952a70e351bfc85 Jul 02 23:10:55.299229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.299210 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" Jul 02 23:10:55.303388 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.303364 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09249fe_61b2_4479_bf24_fcee604ce390.slice/crio-a52c5c8043a685c5e001ec6b6e5fd2e58faf442231a42d746309e7a4b074e0e4 WatchSource:0}: Error finding container a52c5c8043a685c5e001ec6b6e5fd2e58faf442231a42d746309e7a4b074e0e4: Status 404 returned error can't find the container with id a52c5c8043a685c5e001ec6b6e5fd2e58faf442231a42d746309e7a4b074e0e4 Jul 02 23:10:55.308384 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:10:55.308359 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d12fc74_35e0_44d9_96e7_e3c99eb8cc29.slice/crio-9b4c7918501b027c2701a47392da32e7cd607c245bd2655ab96979f95fdf682a WatchSource:0}: Error finding container 9b4c7918501b027c2701a47392da32e7cd607c245bd2655ab96979f95fdf682a: Status 404 returned error can't find the container with id 9b4c7918501b027c2701a47392da32e7cd607c245bd2655ab96979f95fdf682a Jul 02 23:10:55.592688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.592598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:55.592688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.592664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:55.592895 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.592822 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:55.592895 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.592883 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:10:56.592864518 +0000 UTC m=+3.085056173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:55.593322 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.593302 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:55.593428 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.593358 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:10:56.593343186 +0000 UTC m=+3.085534856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:55.693498 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.693460 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:55.693690 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.693672 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:10:55.693739 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.693697 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:10:55.693739 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.693710 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:55.693802 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:55.693768 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:10:56.693750669 +0000 UTC m=+3.185942335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:55.921063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.920822 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 23:05:54 +0000 UTC" deadline="2028-01-08 16:34:30.635508052 +0000 UTC" Jul 02 23:10:55.921063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.920891 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13313h23m34.714621352s" Jul 02 23:10:55.955475 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:55.955202 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:56.083622 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.083551 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwnt8" event={"ID":"8610b55d-ca08-4394-a36c-001cb102480b","Type":"ContainerStarted","Data":"a11a21cac67aed6783f6fb32a687ffcc4f979996813db8abdea2ede1f110d922"} Jul 02 23:10:56.098653 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.098578 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerStarted","Data":"9b4c7918501b027c2701a47392da32e7cd607c245bd2655ab96979f95fdf682a"} Jul 02 23:10:56.118464 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.118366 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lgxxv" event={"ID":"53f9d8b5-07f2-4d5f-93ae-fe9821920914","Type":"ContainerStarted","Data":"762e0f94355583527393081eb8f829df7f14cf6eb85f7cd0a952a70e351bfc85"} Jul 02 23:10:56.137560 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.137434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-twd6k" event={"ID":"4241c9b5-8638-4e3f-a515-8a9862cdf8a2","Type":"ContainerStarted","Data":"aacb458465255890c02c2cd30f3150fdb8b96f913c61e3d4d15eebac95619434"} Jul 02 23:10:56.144029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.143772 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp285" event={"ID":"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13","Type":"ContainerStarted","Data":"eb1f9869c3f2a90b95201aa697ab05f380e9e1f4fa167d943347cc989b714383"} Jul 02 23:10:56.160953 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.160912 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" event={"ID":"46bec173-516c-426e-8db3-dca18369d958","Type":"ContainerStarted","Data":"8d0a9fba938d91b385c890a143f70f25e859e9a8352c9d5e2b30bd7b4a2253b7"} Jul 02 23:10:56.183544 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.183446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5qjns" event={"ID":"248d33c7-27bc-469d-8bbe-53669184b131","Type":"ContainerStarted","Data":"ee8b32f5ba942d1812ea780c4e15d87a7a1b7b37f66a7b440cb4192d284344bc"} Jul 02 23:10:56.201339 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.201301 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"d891c81342273346c0aa567a613298deaee4532f56cb2a46e2a0c45063cdb809"} Jul 02 23:10:56.230456 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.230265 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4br4g" event={"ID":"a09249fe-61b2-4479-bf24-fcee604ce390","Type":"ContainerStarted","Data":"a52c5c8043a685c5e001ec6b6e5fd2e58faf442231a42d746309e7a4b074e0e4"} Jul 02 23:10:56.602719 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.602618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:56.602719 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.602688 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:56.602935 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.602819 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:56.602935 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.602879 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:10:58.602862615 +0000 UTC m=+5.095054270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:56.603052 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.602940 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:56.603052 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.602972 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:10:58.602961389 +0000 UTC m=+5.095153044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:56.686603 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.686362 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:56.704015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.703374 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:56.704015 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.703561 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:10:56.704015 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.703584 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:10:56.704015 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.703596 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:56.704015 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:56.703656 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:10:58.703637328 +0000 UTC m=+5.195828998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:56.921199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.921107 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 23:05:54 +0000 UTC" deadline="2028-02-06 06:42:32.627615494 +0000 UTC" Jul 02 23:10:56.921199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:56.921147 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13999h31m35.706473359s" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:57.063002 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:57.063134 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:57.063567 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:57.063675 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:57.063813 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:57.063962 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:57.063890 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:10:57.982314 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:57.982082 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 23:10:58.621205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:58.621167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:58.621383 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:58.621228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:58.621459 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.621385 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:58.621459 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.621446 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:02.62142893 +0000 UTC m=+9.113620584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:10:58.621907 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.621818 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:58.621907 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.621880 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:11:02.621864624 +0000 UTC m=+9.114056278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:10:58.722244 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:58.722207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:58.722430 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.722404 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:10:58.722430 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.722428 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:10:58.722565 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.722441 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:58.722565 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:58.722500 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:11:02.722482908 +0000 UTC m=+9.214674581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:10:59.062876 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:59.062778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:10:59.062876 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:59.062823 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:10:59.063371 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:59.062936 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:10:59.063371 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:59.063358 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:10:59.063487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:10:59.063413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:10:59.063558 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:10:59.063481 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:01.062807 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:01.062775 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:01.063292 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:01.062904 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:01.063292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:01.062785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:01.063292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:01.062785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:01.063292 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:01.063004 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:01.063292 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:01.063092 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:02.656455 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:02.656279 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:02.656455 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:02.656375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:02.656971 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.656474 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:02.656971 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.656486 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:02.656971 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.656574 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:10.656548772 +0000 UTC m=+17.148740440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:02.656971 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.656610 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:11:10.656594435 +0000 UTC m=+17.148786102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:02.757032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:02.756993 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:02.757217 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.757199 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:11:02.757292 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.757224 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:11:02.757292 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.757238 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:02.757391 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:02.757313 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:11:10.757294507 +0000 UTC m=+17.249486176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:03.062472 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:03.062339 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:03.062687 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:03.062480 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:03.063053 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:03.062813 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:03.063053 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:03.062845 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:03.063053 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:03.062926 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:03.063053 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:03.063005 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:05.062504 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:05.062461 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:05.062926 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:05.062461 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:05.062926 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:05.062487 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:05.062926 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:05.062654 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:05.062926 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:05.062770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:05.062926 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:05.062883 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:07.062780 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:07.062563 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:07.063213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:07.062606 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:07.063213 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:07.062893 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:07.063213 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:07.062986 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:07.063213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:07.062606 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:07.063213 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:07.063079 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:09.062330 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:09.062285 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:09.062785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:09.062310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:09.062785 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:09.062427 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:09.062785 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:09.062455 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:09.062785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:09.062285 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:09.062785 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:09.062525 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:10.722948 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:10.722914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:10.723377 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:10.722969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:10.723377 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.723086 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:10.723377 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.723107 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:10.723377 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.723159 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:11:26.723137717 +0000 UTC m=+33.215329372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:10.723377 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.723179 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:26.723169295 +0000 UTC m=+33.215360948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:10.824158 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:10.824114 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:10.824332 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.824277 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:11:10.824332 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.824306 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:11:10.824332 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.824318 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:10.824454 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:10.824385 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:11:26.824367895 +0000 UTC m=+33.316559549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:11.062205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:11.062165 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:11.062375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:11.062289 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:11.062375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:11.062303 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:11.062375 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:11.062298 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:11.062561 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:11.062415 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:11.062561 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:11.062497 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:13.062897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:13.062849 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:13.062897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:13.062879 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:13.062897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:13.062901 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:13.063450 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:13.062991 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:13.063450 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:13.063067 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:13.063450 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:13.063146 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:15.062330 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.062301 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:15.062330 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.062327 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:15.062718 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:15.062416 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:15.062718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.062467 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:15.062718 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:15.062572 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:15.062718 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:15.062627 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:15.276462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.276431 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-twd6k" event={"ID":"4241c9b5-8638-4e3f-a515-8a9862cdf8a2","Type":"ContainerStarted","Data":"42413f050cc775b57bf187127ec34fff797259ca15cf596a3b5ea7c1a2ccb42b"} Jul 02 23:11:15.278323 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.278279 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" event={"ID":"b956c94c524f1281d984172ee6c41e32","Type":"ContainerStarted","Data":"d9a76bb2f5be04cc9a3c423f69fdcca41b5e4e9f006442a5c314d8778674679d"} Jul 02 23:11:15.279839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.279816 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4br4g" event={"ID":"a09249fe-61b2-4479-bf24-fcee604ce390","Type":"ContainerStarted","Data":"9af8f65f9fc07ffcc4f2aebff8c0a60e38f96a21c16cee1e8afd221e3eeb3dbf"} Jul 02 23:11:15.305714 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.305657 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-twd6k" podStartSLOduration=1.723295442 podStartE2EDuration="21.305634857s" totalImagesPullingTime="19.582339415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:15.303713264 +0000 UTC m=+21.795904944" watchObservedRunningTime="2026-07-02 23:11:15.305634857 +0000 UTC m=+21.797826532" Jul 02 23:11:15.327336 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.327240 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4br4g" podStartSLOduration=1.779678233 podStartE2EDuration="21.326520906s" totalImagesPullingTime="19.546842673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:15.32494978 +0000 UTC m=+21.817141468" watchObservedRunningTime="2026-07-02 23:11:15.326520906 +0000 UTC m=+21.818712582" Jul 02 23:11:15.341990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:15.341620 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-16.ec2.internal" podStartSLOduration=21.34160178 podStartE2EDuration="21.34160178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:15.341312494 +0000 UTC m=+21.833504172" watchObservedRunningTime="2026-07-02 23:11:15.34160178 +0000 UTC m=+21.833793456" Jul 02 23:11:16.284358 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.283944 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="7895102d84c5ad88e37a5924b5ed38d89725528c309eb4ef3d612441ee0eb7a0" exitCode=0 Jul 02 23:11:16.285069 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.284050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"7895102d84c5ad88e37a5924b5ed38d89725528c309eb4ef3d612441ee0eb7a0"} Jul 02 23:11:16.286213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.286159 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vp285" event={"ID":"b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13","Type":"ContainerStarted","Data":"e05af5b4e5c2e3f0d105b1a91a26a91c34f345c133b72ed4b79a1dec30e8b22e"} Jul 02 23:11:16.287961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.287931 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" event={"ID":"46bec173-516c-426e-8db3-dca18369d958","Type":"ContainerStarted","Data":"354ff0df8bfcf1b087e1d387ab451628c8da2ab047060342478dab2aeea8adeb"} Jul 02 23:11:16.289587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.289561 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5qjns" event={"ID":"248d33c7-27bc-469d-8bbe-53669184b131","Type":"ContainerStarted","Data":"7de77587404e677b8c8fa161758a9b0da83a232e57d83e9658a0e4ef580b2e98"} Jul 02 23:11:16.293151 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293107 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"70390863b4de9c70619ac73735c53c21728c68d6f6f59754abe67107ab5e84b7"} Jul 02 23:11:16.293262 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"bbf9cf3a78bccba06fa9f70354c7e3a32ca63d34040bbcec8e1a4b4fb47a8670"} Jul 02 23:11:16.293262 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293232 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"fb25d05643426509f35e84a81725cd328b279aa679a62bfa8403cedd182b0037"} Jul 02 23:11:16.293395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"a8b580c2698f09fd9ed20f2ed98cd89d5afd7649c00975fc982f25d83c7ae738"} Jul 02 23:11:16.293395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293335 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"ee95e982b69b38c0d0ac9011a4300562e73ea0995709010e6f4412571c70932a"} Jul 02 23:11:16.293395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.293381 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"14500dfc375acfae332981cc99c473a7a74c1a72dab4fcbc428d9f7a099a69fb"} Jul 02 23:11:16.294672 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.294647 2561 generic.go:356] "Generic (PLEG): container finished" podID="27ac7136b72eeb4c1bec7700ebb2a216" containerID="ff82cfcca6b009ac52ae8da7521d31f0b511229d8147b79be4d244e297818c89" exitCode=0 Jul 02 23:11:16.294769 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.294735 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" event={"ID":"27ac7136b72eeb4c1bec7700ebb2a216","Type":"ContainerDied","Data":"ff82cfcca6b009ac52ae8da7521d31f0b511229d8147b79be4d244e297818c89"} Jul 02 23:11:16.296987 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.296963 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwnt8" event={"ID":"8610b55d-ca08-4394-a36c-001cb102480b","Type":"ContainerStarted","Data":"e0269d5c42467192a0b79b85e11a527de6fdf13de98a79e95461bed41027acf6"} Jul 02 23:11:16.323543 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.323494 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vp285" podStartSLOduration=7.507782525 podStartE2EDuration="22.323478787s" totalImagesPullingTime="14.815696262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:16.323179056 +0000 UTC m=+22.815370731" watchObservedRunningTime="2026-07-02 23:11:16.323478787 +0000 UTC m=+22.815670461" Jul 02 23:11:16.355355 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.355297 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5qjns" podStartSLOduration=2.725802221 podStartE2EDuration="22.355275687s" totalImagesPullingTime="19.629473466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:16.354834177 +0000 UTC m=+22.847025855" watchObservedRunningTime="2026-07-02 23:11:16.355275687 +0000 UTC m=+22.847467364" Jul 02 23:11:16.355617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:16.355524 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwnt8" podStartSLOduration=7.556108531 podStartE2EDuration="22.35551403s" totalImagesPullingTime="14.799405499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:16.338916205 +0000 UTC m=+22.831107883" watchObservedRunningTime="2026-07-02 23:11:16.35551403 +0000 UTC m=+22.847705707" Jul 02 23:11:17.062929 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.062897 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:17.063085 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.062897 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:17.063085 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.063009 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:17.063085 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:17.063011 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:17.063085 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:17.063079 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:17.063222 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:17.063196 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:17.093307 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.093281 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 23:11:17.272647 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.272605 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:17.301396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.301362 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" event={"ID":"27ac7136b72eeb4c1bec7700ebb2a216","Type":"ContainerStarted","Data":"fc93fca977a2ae3b3fdfb1029ce4c9efd0668c8dea79b5cadc911df3e263552f"} Jul 02 23:11:17.303042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.303013 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lgxxv" event={"ID":"53f9d8b5-07f2-4d5f-93ae-fe9821920914","Type":"ContainerStarted","Data":"462d5d77fce693ded9e5ef02092817f0472159c6e8a895dffce2d5abbacaf564"} Jul 02 23:11:17.305050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.305018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" event={"ID":"46bec173-516c-426e-8db3-dca18369d958","Type":"ContainerStarted","Data":"0cb8babc6a15b3d54c2baecd930da5b266769346f63f5772f5ded54b95bfe768"} Jul 02 23:11:17.337753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.337689 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lgxxv" podStartSLOduration=3.7858407659999997 podStartE2EDuration="23.337667297s" totalImagesPullingTime="19.551826531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:17.337170386 +0000 UTC m=+23.829362071" watchObservedRunningTime="2026-07-02 23:11:17.337667297 +0000 UTC m=+23.829858974" Jul 02 23:11:17.337940 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.337771 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-16.ec2.internal" podStartSLOduration=23.337766157 podStartE2EDuration="23.337766157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:17.320973402 +0000 UTC m=+23.813165091" watchObservedRunningTime="2026-07-02 23:11:17.337766157 +0000 UTC m=+23.829957831" Jul 02 23:11:17.517347 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.517302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:11:17.517996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.517979 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:11:17.833945 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.833909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:17.971107 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.971058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:17.979219 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.979087 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T23:11:17.093298919Z","UUID":"2fb63fb7-e4c5-4485-a71c-875c8d389572","Handler":null,"Name":"","Endpoint":""} Jul 02 23:11:17.980829 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.980808 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 23:11:17.980965 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:17.980853 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 23:11:18.071119 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.071087 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:18.168347 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.168315 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:18.275765 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.275732 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:18.311590 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.311560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"c3efee24fa724ce76a6dc5b4faa1c142e7593536d0b3ce173ca3710cd9425ef4"} Jul 02 23:11:18.312144 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.311990 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:11:18.312376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.312358 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5qjns" Jul 02 23:11:18.444557 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:18.444520 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:19.062120 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.062087 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:19.062288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.062199 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:19.062288 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:19.062208 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:19.062288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.062217 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:19.062452 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:19.062316 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:19.062452 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:19.062398 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:19.316456 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.316371 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" event={"ID":"46bec173-516c-426e-8db3-dca18369d958","Type":"ContainerStarted","Data":"3cc123d8c9451d29cd2804ebd060c1d3b2fb247d258ff2878932d24f7457b1d7"} Jul 02 23:11:19.335736 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.335679 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v57s8" podStartSLOduration=3.309866214 podStartE2EDuration="25.335657021s" totalImagesPullingTime="22.025790807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:19.335522046 +0000 UTC m=+25.827713723" watchObservedRunningTime="2026-07-02 23:11:19.335657021 +0000 UTC m=+25.827848699" Jul 02 23:11:19.447386 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.447351 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:19.834665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:19.834622 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:20.035930 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:20.035884 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:20.233196 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:20.233148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:20.436506 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:20.436417 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:21.049223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.049043 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:21.061965 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.061940 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:21.062131 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.061973 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:21.062131 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.061941 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:21.062131 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:21.062040 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:21.062285 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:21.062143 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:21.062285 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:21.062245 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:21.157345 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.157314 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:21.324579 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.324472 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" event={"ID":"eec0b321-ccf0-4210-8f24-0b1061c43743","Type":"ContainerStarted","Data":"fe453919d43b0017c246fbabf91920ce032f59670212b5aeaf8198cd6dcbae94"} Jul 02 23:11:21.324816 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.324792 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:21.326397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.326366 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="4766cd38b3c24b8ecb4b1e41967471992ab2728999800604dbccb4353c12d15b" exitCode=0 Jul 02 23:11:21.326492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.326426 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"4766cd38b3c24b8ecb4b1e41967471992ab2728999800604dbccb4353c12d15b"} Jul 02 23:11:21.340720 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.340691 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:21.353748 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.353706 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" podStartSLOduration=7.335300732 podStartE2EDuration="27.353691941s" totalImagesPullingTime="20.018391209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:21.353155595 +0000 UTC m=+27.845347294" watchObservedRunningTime="2026-07-02 23:11:21.353691941 +0000 UTC m=+27.845883616" Jul 02 23:11:21.440579 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.440515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:21.505111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.505076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:21.639993 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:21.639907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.035159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.035129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.240492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.240413 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.305804 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.305772 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.330918 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.330884 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="7fdc01d5ff92f2b645df20110233970e56098033d92bb459527f1c1bb130d2da" exitCode=0 Jul 02 23:11:22.331079 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.330956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"7fdc01d5ff92f2b645df20110233970e56098033d92bb459527f1c1bb130d2da"} Jul 02 23:11:22.331415 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.331395 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:22.331551 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.331470 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:22.346979 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.346955 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:22.389435 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.389402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.461352 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.461319 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.525221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.525181 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.679554 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.679510 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.854425 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.854345 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:22.918418 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:22.918390 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.058881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.058854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.062869 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.062852 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:23.062956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.062943 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:23.062999 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:23.062952 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:23.062999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.062968 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:23.063055 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:23.063043 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:23.063123 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:23.063107 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:23.255956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.255927 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.310359 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.310330 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.437282 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.437252 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.530611 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.530580 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.644577 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.644545 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:23.840022 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:23.839938 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:24.036201 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.036012 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:24.136599 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.136509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:24.237295 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.237257 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:24.337824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.337793 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:24.338375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.338353 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="62a0523d9e974e9d630333675f0358e407cbcf840474f3e5de0d2c19a54eec0c" exitCode=0 Jul 02 23:11:24.338469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.338446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"62a0523d9e974e9d630333675f0358e407cbcf840474f3e5de0d2c19a54eec0c"} Jul 02 23:11:24.837816 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:24.837779 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:25.062273 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.062242 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:25.062462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.062243 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:25.062462 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:25.062376 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:25.062462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.062257 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:25.062620 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:25.062454 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:25.062667 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:25.062612 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:25.086682 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.086641 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:25.187996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.187909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:25.288256 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.288222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:25.431357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.431320 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:25.483811 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:25.483728 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:26.053633 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.053598 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:26.636493 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.636457 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:26.744688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.744652 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:26.744877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.744705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:26.744877 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.744830 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:26.744877 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.744831 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:26.744996 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.744892 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs podName:1f51cd4b-2792-414b-b9c0-54dcb8e390c0 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:58.744872502 +0000 UTC m=+65.237064154 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs") pod "network-metrics-daemon-hf4vd" (UID: "1f51cd4b-2792-414b-b9c0-54dcb8e390c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 23:11:26.744996 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.744912 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret podName:a7cc66b4-608b-4631-827a-2c764e4e023a nodeName:}" failed. No retries permitted until 2026-07-02 23:11:58.744902227 +0000 UTC m=+65.237093884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret") pod "global-pull-secret-syncer-8tm84" (UID: "a7cc66b4-608b-4631-827a-2c764e4e023a") : object "kube-system"/"original-pull-secret" not registered Jul 02 23:11:26.838023 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.837989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:26.846040 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:26.846007 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:26.846232 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.846213 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 23:11:26.846276 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.846239 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 23:11:26.846276 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.846253 2561 projected.go:196] Error preparing data for projected volume kube-api-access-bwrxr for pod openshift-network-diagnostics/network-check-target-c2pwk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:26.846361 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:26.846320 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr podName:06152e08-c5a4-4607-92df-af45954f43ba nodeName:}" failed. No retries permitted until 2026-07-02 23:11:58.846301104 +0000 UTC m=+65.338492761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwrxr" (UniqueName: "kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr") pod "network-check-target-c2pwk" (UID: "06152e08-c5a4-4607-92df-af45954f43ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 23:11:27.035737 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:27.035699 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:27.062912 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:27.062874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:27.063369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:27.062991 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:27.063369 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:27.063004 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:27.063369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:27.063026 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:27.063369 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:27.063123 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:27.063369 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:27.063214 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:27.254658 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:27.254624 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:28.039715 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:28.039664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:28.436413 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:28.436338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:28.639305 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:28.639270 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:28.832042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:28.832005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:29.062079 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:29.062042 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:29.062245 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:29.062151 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:29.062245 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:29.062175 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:29.062245 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:29.062168 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:29.062364 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:29.062246 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:29.062364 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:29.062335 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:29.352739 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:29.352705 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="dbcb10b8bac659e6c48e1fb8631780ad91f652eafe90c32d036ee14b4d68127e" exitCode=0 Jul 02 23:11:29.352896 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:29.352764 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"dbcb10b8bac659e6c48e1fb8631780ad91f652eafe90c32d036ee14b4d68127e"} Jul 02 23:11:30.358043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:30.358007 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d12fc74-35e0-44d9-96e7-e3c99eb8cc29" containerID="42f2fcaf7249403819a495054f23c85314751fbe3dc62fc42183427b37cfe2a8" exitCode=0 Jul 02 23:11:30.358422 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:30.358057 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerDied","Data":"42f2fcaf7249403819a495054f23c85314751fbe3dc62fc42183427b37cfe2a8"} Jul 02 23:11:31.062505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:31.062472 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:31.062505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:31.062506 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:31.062747 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:31.062601 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:31.062747 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:31.062606 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:31.062747 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:31.062670 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:31.062856 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:31.062747 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:31.363231 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:31.363145 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" event={"ID":"1d12fc74-35e0-44d9-96e7-e3c99eb8cc29","Type":"ContainerStarted","Data":"945ce99c14f7a9bfd99963d39aa2c78a9067c86b1ee499c1b7dcc1bdfd9f2149"} Jul 02 23:11:31.389005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:31.388956 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pmmhc" podStartSLOduration=7.278197175 podStartE2EDuration="37.388940125s" totalImagesPullingTime="30.11074295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:31.388550049 +0000 UTC m=+37.880741718" watchObservedRunningTime="2026-07-02 23:11:31.388940125 +0000 UTC m=+37.881131800" Jul 02 23:11:33.062665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:33.062631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:33.063117 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:33.062631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:33.063117 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:33.062734 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:33.063117 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:33.062631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:33.063117 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:33.062807 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:33.063117 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:33.062893 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:33.828223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:33.828183 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:33.992292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:33.992259 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.184552 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.184455 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.291807 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.291773 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.517526 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.517500 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.614228 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.614193 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.720478 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.720441 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.816018 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.815831 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:34.912931 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.912899 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hf4vd"] Jul 02 23:11:34.913104 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.913066 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:34.913208 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:34.913188 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:34.915660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.915519 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tm84"] Jul 02 23:11:34.915660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.915646 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:34.915838 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:34.915745 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:34.916213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.916174 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c2pwk"] Jul 02 23:11:34.916338 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:34.916267 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:34.916414 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:34.916352 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:35.128417 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:35.128325 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:35.202260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:35.202233 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:36.067895 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:36.067859 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:36.134769 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:36.134733 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:37.062581 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:37.062527 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:37.062994 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:37.062555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:37.062994 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:37.062643 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c2pwk" podUID="06152e08-c5a4-4607-92df-af45954f43ba" Jul 02 23:11:37.062994 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:37.062681 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:37.062994 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:37.062784 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hf4vd" podUID="1f51cd4b-2792-414b-b9c0-54dcb8e390c0" Jul 02 23:11:37.062994 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:37.062819 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tm84" podUID="a7cc66b4-608b-4631-827a-2c764e4e023a" Jul 02 23:11:38.120418 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.120386 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:38.235736 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.235709 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:38.735927 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.735899 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-16.ec2.internal" event="NodeReady" Jul 02 23:11:38.736094 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.736027 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 23:11:38.815666 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.815629 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rv29q"] Jul 02 23:11:38.828106 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.828081 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qb68g"] Jul 02 23:11:38.828226 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.828207 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.832678 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.832657 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7j8n\"" Jul 02 23:11:38.833608 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.833591 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 23:11:38.833677 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.833630 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 23:11:38.833721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.833598 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 23:11:38.835644 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.835630 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 23:11:38.838128 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.838112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:38.840891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.840868 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 23:11:38.840891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.840883 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 23:11:38.841018 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.840916 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jlfnl\"" Jul 02 23:11:38.849194 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.849169 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rv29q"] Jul 02 23:11:38.850402 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.850379 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qb68g"] Jul 02 23:11:38.921072 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.921005 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-szfsv"] Jul 02 23:11:38.933441 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-data-volume\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.933558 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933470 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4tv\" (UniqueName: \"kubernetes.io/projected/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-api-access-7w4tv\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.933558 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3559d7-7419-46ed-b51a-d7ec3d69da97-tmp-dir\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:38.933639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933578 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3559d7-7419-46ed-b51a-d7ec3d69da97-metrics-tls\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:38.933639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933608 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr4dz\" (UniqueName: \"kubernetes.io/projected/db3559d7-7419-46ed-b51a-d7ec3d69da97-kube-api-access-sr4dz\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:38.933639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-crio-socket\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.933725 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933704 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3559d7-7419-46ed-b51a-d7ec3d69da97-config-volume\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:38.933759 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933731 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.933759 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.933748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:38.942100 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.942071 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szfsv"] Jul 02 23:11:38.942244 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.942195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:38.945718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.945692 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 23:11:38.945862 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.945756 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 23:11:38.945936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.945899 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 23:11:38.945995 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:38.945938 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s62t9\"" Jul 02 23:11:39.034767 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034731 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3559d7-7419-46ed-b51a-d7ec3d69da97-metrics-tls\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.034767 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034765 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr4dz\" (UniqueName: \"kubernetes.io/projected/db3559d7-7419-46ed-b51a-d7ec3d69da97-kube-api-access-sr4dz\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034785 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-crio-socket\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034825 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3559d7-7419-46ed-b51a-d7ec3d69da97-config-volume\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034851 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034873 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-data-volume\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.034982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034953 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4tv\" (UniqueName: \"kubernetes.io/projected/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-api-access-7w4tv\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.035182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.034984 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3559d7-7419-46ed-b51a-d7ec3d69da97-tmp-dir\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.035182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.035054 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-crio-socket\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.035365 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.035344 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-data-volume\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.035403 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.035367 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3559d7-7419-46ed-b51a-d7ec3d69da97-tmp-dir\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.035587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.035527 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.035587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.035588 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3559d7-7419-46ed-b51a-d7ec3d69da97-config-volume\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.039068 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.039035 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3559d7-7419-46ed-b51a-d7ec3d69da97-metrics-tls\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.039175 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.039035 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.045911 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.045885 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4tv\" (UniqueName: \"kubernetes.io/projected/09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6-kube-api-access-7w4tv\") pod \"insights-runtime-extractor-rv29q\" (UID: \"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6\") " pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.046020 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.045959 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr4dz\" (UniqueName: \"kubernetes.io/projected/db3559d7-7419-46ed-b51a-d7ec3d69da97-kube-api-access-sr4dz\") pod \"dns-default-qb68g\" (UID: \"db3559d7-7419-46ed-b51a-d7ec3d69da97\") " pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.062422 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.062394 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:39.062587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.062394 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:39.062587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.062394 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:39.067985 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.067965 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmk5n\"" Jul 02 23:11:39.068134 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.068065 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 23:11:39.068317 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.068304 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 23:11:39.068392 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.068372 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prtl8\"" Jul 02 23:11:39.069165 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.069149 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 23:11:39.069268 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.069222 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 23:11:39.116292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.116264 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:39.136294 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.136260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9526752-62a6-4815-8d5e-0a523d822544-cert\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.136712 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.136374 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gzk4\" (UniqueName: \"kubernetes.io/projected/f9526752-62a6-4815-8d5e-0a523d822544-kube-api-access-9gzk4\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.137192 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.137176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rv29q" Jul 02 23:11:39.145486 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.145464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:39.215578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.215499 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:39.237810 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.237704 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzk4\" (UniqueName: \"kubernetes.io/projected/f9526752-62a6-4815-8d5e-0a523d822544-kube-api-access-9gzk4\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.237810 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.237761 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9526752-62a6-4815-8d5e-0a523d822544-cert\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.240331 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.240300 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9526752-62a6-4815-8d5e-0a523d822544-cert\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.247236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.247207 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzk4\" (UniqueName: \"kubernetes.io/projected/f9526752-62a6-4815-8d5e-0a523d822544-kube-api-access-9gzk4\") pod \"ingress-canary-szfsv\" (UID: \"f9526752-62a6-4815-8d5e-0a523d822544\") " pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.250966 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.250936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-szfsv" Jul 02 23:11:39.361080 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.361058 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rv29q"] Jul 02 23:11:39.364788 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:39.364606 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cc0ae0_cdf9_4859_b5ec_d0c0a923f0b6.slice/crio-48eb110601ef886a615bfed4f047d06388a5ac15da2aed0ee25d73c1dcc56ac2 WatchSource:0}: Error finding container 48eb110601ef886a615bfed4f047d06388a5ac15da2aed0ee25d73c1dcc56ac2: Status 404 returned error can't find the container with id 48eb110601ef886a615bfed4f047d06388a5ac15da2aed0ee25d73c1dcc56ac2 Jul 02 23:11:39.366347 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.366320 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qb68g"] Jul 02 23:11:39.370144 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:39.370119 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3559d7_7419_46ed_b51a_d7ec3d69da97.slice/crio-255bb1da478ff55e1200281bcb7565466f6f83eeea374a3c2c3334dac1e36036 WatchSource:0}: Error finding container 255bb1da478ff55e1200281bcb7565466f6f83eeea374a3c2c3334dac1e36036: Status 404 returned error can't find the container with id 255bb1da478ff55e1200281bcb7565466f6f83eeea374a3c2c3334dac1e36036 Jul 02 23:11:39.382613 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.382520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rv29q" event={"ID":"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6","Type":"ContainerStarted","Data":"48eb110601ef886a615bfed4f047d06388a5ac15da2aed0ee25d73c1dcc56ac2"} Jul 02 23:11:39.384037 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.384011 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qb68g" event={"ID":"db3559d7-7419-46ed-b51a-d7ec3d69da97","Type":"ContainerStarted","Data":"255bb1da478ff55e1200281bcb7565466f6f83eeea374a3c2c3334dac1e36036"} Jul 02 23:11:39.415188 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:39.415155 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-szfsv"] Jul 02 23:11:39.419083 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:39.419052 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9526752_62a6_4815_8d5e_0a523d822544.slice/crio-b985d2912c806881e43410426972ff19dd7fff04d0dc12cbb3bbb74fa2689c5c WatchSource:0}: Error finding container b985d2912c806881e43410426972ff19dd7fff04d0dc12cbb3bbb74fa2689c5c: Status 404 returned error can't find the container with id b985d2912c806881e43410426972ff19dd7fff04d0dc12cbb3bbb74fa2689c5c Jul 02 23:11:40.290589 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:40.290505 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:40.352266 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:40.352180 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:40.389420 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:40.389339 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szfsv" event={"ID":"f9526752-62a6-4815-8d5e-0a523d822544","Type":"ContainerStarted","Data":"b985d2912c806881e43410426972ff19dd7fff04d0dc12cbb3bbb74fa2689c5c"} Jul 02 23:11:41.640906 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:41.640875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:41.716021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:41.715987 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:43.729523 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.729468 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:43.746645 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.746603 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:11:43.748739 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.748715 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.753933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.753889 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 23:11:43.754087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754002 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnwk9\"" Jul 02 23:11:43.754087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754052 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 23:11:43.754213 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754144 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 23:11:43.754268 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754004 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 23:11:43.754268 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754222 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 23:11:43.754360 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.754339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 23:11:43.755552 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.755487 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 23:11:43.761505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.761478 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:11:43.772878 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.772841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.773042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.772889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.773042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.772953 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.773042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.772975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.773042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.772991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dbx\" (UniqueName: \"kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.773216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.773111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.804571 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.804515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:43.874201 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.874369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874221 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dbx\" (UniqueName: \"kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.874369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.874369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.874518 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874404 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.874518 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.874449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.875267 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.875171 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.875267 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.875177 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.875267 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.875231 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.877895 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.877872 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.877998 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.877873 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:43.886615 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:43.886588 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dbx\" (UniqueName: \"kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx\") pod \"console-58d697c4b-mr79q\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:44.061409 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.061358 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:44.493130 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.493093 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:44.610802 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.610771 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:44.692062 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.691993 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8"] Jul 02 23:11:44.695050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.695019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:44.698252 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.698230 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 02 23:11:44.698367 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.698301 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l9wf9\"" Jul 02 23:11:44.707578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.707548 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8"] Jul 02 23:11:44.712777 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.712744 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:44.782386 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.782335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdfe9b5e-941f-4755-a3d1-a658a18dd443-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r8tc8\" (UID: \"cdfe9b5e-941f-4755-a3d1-a658a18dd443\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:44.884116 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.883893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdfe9b5e-941f-4755-a3d1-a658a18dd443-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r8tc8\" (UID: \"cdfe9b5e-941f-4755-a3d1-a658a18dd443\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:44.892544 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.892028 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cdfe9b5e-941f-4755-a3d1-a658a18dd443-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-r8tc8\" (UID: \"cdfe9b5e-941f-4755-a3d1-a658a18dd443\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:44.918308 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.918269 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:11:44.929888 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:44.929857 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95d679f_356a_46ee_a1ff_abab7f1b9f51.slice/crio-4ea467b39c3e76fc77231e09a591fe90dcb8b475d2e45f670d77ba2852bf4309 WatchSource:0}: Error finding container 4ea467b39c3e76fc77231e09a591fe90dcb8b475d2e45f670d77ba2852bf4309: Status 404 returned error can't find the container with id 4ea467b39c3e76fc77231e09a591fe90dcb8b475d2e45f670d77ba2852bf4309 Jul 02 23:11:44.950439 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:44.950404 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:45.006827 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.006796 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:45.131982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.131745 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:11:45.134346 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.134329 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.143716 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.143687 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 23:11:45.148052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.148017 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:45.148906 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.148886 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8"] Jul 02 23:11:45.152452 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.152430 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:11:45.153164 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:45.153142 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfe9b5e_941f_4755_a3d1_a658a18dd443.slice/crio-c36af5cc63b115378bd582745ad4d386c42cfac62550752f4b8fd9550136206a WatchSource:0}: Error finding container c36af5cc63b115378bd582745ad4d386c42cfac62550752f4b8fd9550136206a: Status 404 returned error can't find the container with id c36af5cc63b115378bd582745ad4d386c42cfac62550752f4b8fd9550136206a Jul 02 23:11:45.186221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186192 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186224 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186476 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186476 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186476 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186608 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.186608 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.186513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btkj\" (UniqueName: \"kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.264543 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.264497 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:45.287000 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.286968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287163 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287163 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287048 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287163 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btkj\" (UniqueName: \"kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287359 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.287410 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.288108 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287837 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.288108 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.287849 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.288108 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.288022 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.288333 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.288285 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.291824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.291794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.291971 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.291955 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.299708 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.299684 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btkj\" (UniqueName: \"kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj\") pod \"console-9b49d8864-9gzk7\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.402960 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.402929 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qb68g" event={"ID":"db3559d7-7419-46ed-b51a-d7ec3d69da97","Type":"ContainerStarted","Data":"a47067b79b8bf96f22ac9e4ed7006075db4fc1cc2be5731f3d16f8e517221b04"} Jul 02 23:11:45.403053 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.403018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qb68g" event={"ID":"db3559d7-7419-46ed-b51a-d7ec3d69da97","Type":"ContainerStarted","Data":"f482b60343e53ad8beab5075373949ed8d15739e78aa2fe711fc65c6fbb6a6e6"} Jul 02 23:11:45.403053 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.403049 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:45.404392 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.404371 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-szfsv" event={"ID":"f9526752-62a6-4815-8d5e-0a523d822544","Type":"ContainerStarted","Data":"ed3dc938f841d994b2a7fd10031067feebdcc6af4e71e822f3e75f1bf67e04ca"} Jul 02 23:11:45.405684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.405662 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rv29q" event={"ID":"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6","Type":"ContainerStarted","Data":"ef4532cbbf7e417c197aa6c8bfbe602e7cf7c76f80d644f9b03add67a1aaa10a"} Jul 02 23:11:45.406747 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.406719 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" event={"ID":"cdfe9b5e-941f-4755-a3d1-a658a18dd443","Type":"ContainerStarted","Data":"c36af5cc63b115378bd582745ad4d386c42cfac62550752f4b8fd9550136206a"} Jul 02 23:11:45.407858 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.407834 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d697c4b-mr79q" event={"ID":"d95d679f-356a-46ee-a1ff-abab7f1b9f51","Type":"ContainerStarted","Data":"4ea467b39c3e76fc77231e09a591fe90dcb8b475d2e45f670d77ba2852bf4309"} Jul 02 23:11:45.428829 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.428779 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qb68g" podStartSLOduration=2.053859012 podStartE2EDuration="7.428758887s" totalImagesPullingTime="5.374899875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:45.425890491 +0000 UTC m=+51.918082167" watchObservedRunningTime="2026-07-02 23:11:45.428758887 +0000 UTC m=+51.920950565" Jul 02 23:11:45.444227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.444198 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:45.448624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.447448 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-szfsv" podStartSLOduration=2.117625692 podStartE2EDuration="7.447430127s" totalImagesPullingTime="5.329804435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:45.446695942 +0000 UTC m=+51.938887618" watchObservedRunningTime="2026-07-02 23:11:45.447430127 +0000 UTC m=+51.939621797" Jul 02 23:11:45.517103 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.517026 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:45.598060 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.598029 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:11:45.601979 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:45.601952 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233b1304_733c_410d_acce_498fe0a7fd6e.slice/crio-7a1bf7bda3c390d6fd4767d6cc1f55323baac85f89f32cb25f78f53fddcfdb4c WatchSource:0}: Error finding container 7a1bf7bda3c390d6fd4767d6cc1f55323baac85f89f32cb25f78f53fddcfdb4c: Status 404 returned error can't find the container with id 7a1bf7bda3c390d6fd4767d6cc1f55323baac85f89f32cb25f78f53fddcfdb4c Jul 02 23:11:45.603380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:45.603177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.414439 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.414368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b49d8864-9gzk7" event={"ID":"233b1304-733c-410d-acce-498fe0a7fd6e","Type":"ContainerStarted","Data":"7a1bf7bda3c390d6fd4767d6cc1f55323baac85f89f32cb25f78f53fddcfdb4c"} Jul 02 23:11:46.417190 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.417120 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rv29q" event={"ID":"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6","Type":"ContainerStarted","Data":"547ec76440f5b5b8b4c202986acad36b219f46c7d1cb0cb37dd850d38f5f6437"} Jul 02 23:11:46.419202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.419175 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" event={"ID":"cdfe9b5e-941f-4755-a3d1-a658a18dd443","Type":"ContainerStarted","Data":"2aa86c0cfac422ac31d627e70f9152bc7102cd9e281e4789aca8c6978bf881de"} Jul 02 23:11:46.419753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.419732 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:46.425292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.425266 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" Jul 02 23:11:46.436382 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.436324 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-r8tc8" podStartSLOduration=1.342297535 podStartE2EDuration="2.436303919s" totalImagesPullingTime="1.094006384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:46.436078359 +0000 UTC m=+52.928270035" watchObservedRunningTime="2026-07-02 23:11:46.436303919 +0000 UTC m=+52.928495595" Jul 02 23:11:46.541014 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.540887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.597780 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.597730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.707287 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.707247 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.746427 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.746396 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl"] Jul 02 23:11:46.748953 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.748929 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.752130 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.752105 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8fjwb\"" Jul 02 23:11:46.752361 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.752339 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 23:11:46.753715 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.753521 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 23:11:46.753715 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.753598 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 23:11:46.754667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.754060 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 23:11:46.754667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.754064 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 23:11:46.761932 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.761868 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl"] Jul 02 23:11:46.783006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.782974 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.797959 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.797836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.797959 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.797908 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.797959 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.797937 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.798226 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.797982 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljwq\" (UniqueName: \"kubernetes.io/projected/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-kube-api-access-5ljwq\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.854601 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.854504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:46.899045 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.899008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.899212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.899064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.899212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.899121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljwq\" (UniqueName: \"kubernetes.io/projected/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-kube-api-access-5ljwq\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.899212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.899198 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.900347 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.900277 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.904785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.904285 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.904785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.904744 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:46.909143 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:46.909101 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljwq\" (UniqueName: \"kubernetes.io/projected/7bf0327a-1414-4caf-bf9f-d2beaa09d07c-kube-api-access-5ljwq\") pod \"prometheus-operator-567f7b8b7c-d5shl\" (UID: \"7bf0327a-1414-4caf-bf9f-d2beaa09d07c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:47.062623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:47.062502 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" Jul 02 23:11:47.227297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:47.226928 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl"] Jul 02 23:11:47.519706 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:47.519671 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:47.575701 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:47.575632 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:48.147342 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:48.147283 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:48.202781 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:48.202730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:48.286904 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:48.286868 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf0327a_1414_4caf_bf9f_d2beaa09d07c.slice/crio-5ec6dfddcb77f26958cdd6668fb7000b2d077573b0e63a2f5cae0e3690b31388 WatchSource:0}: Error finding container 5ec6dfddcb77f26958cdd6668fb7000b2d077573b0e63a2f5cae0e3690b31388: Status 404 returned error can't find the container with id 5ec6dfddcb77f26958cdd6668fb7000b2d077573b0e63a2f5cae0e3690b31388 Jul 02 23:11:48.431037 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:48.430955 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" event={"ID":"7bf0327a-1414-4caf-bf9f-d2beaa09d07c","Type":"ContainerStarted","Data":"5ec6dfddcb77f26958cdd6668fb7000b2d077573b0e63a2f5cae0e3690b31388"} Jul 02 23:11:48.566870 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:48.566835 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:48.622820 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:48.622786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:50.441266 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.441214 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rv29q" event={"ID":"09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6","Type":"ContainerStarted","Data":"5e6531edb1e818492f7afae301f565a798b1924b20406d3346caea7b6c711137"} Jul 02 23:11:50.443779 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.443743 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d697c4b-mr79q" event={"ID":"d95d679f-356a-46ee-a1ff-abab7f1b9f51","Type":"ContainerStarted","Data":"ba875863860d7a352d711ad0738b45241beecfac4e88edc0b5581b9e602ede03"} Jul 02 23:11:50.446724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.446697 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" event={"ID":"7bf0327a-1414-4caf-bf9f-d2beaa09d07c","Type":"ContainerStarted","Data":"d2f748895b9241c66b55f1105f1d65a70038660e0437ec8650ee59d5616ae94b"} Jul 02 23:11:50.446883 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.446753 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" event={"ID":"7bf0327a-1414-4caf-bf9f-d2beaa09d07c","Type":"ContainerStarted","Data":"1145986e38ba195c9264e49bcc24533571a4a2a2a88d38a07500019cd9f8dfdb"} Jul 02 23:11:50.448654 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.448624 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b49d8864-9gzk7" event={"ID":"233b1304-733c-410d-acce-498fe0a7fd6e","Type":"ContainerStarted","Data":"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229"} Jul 02 23:11:50.463894 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.463834 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rv29q" podStartSLOduration=2.551718189 podStartE2EDuration="12.463816751s" totalImagesPullingTime="9.912098562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:11:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:50.461716672 +0000 UTC m=+56.953908348" watchObservedRunningTime="2026-07-02 23:11:50.463816751 +0000 UTC m=+56.956008426" Jul 02 23:11:50.482814 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.482764 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58d697c4b-mr79q" podStartSLOduration=2.995591711 podStartE2EDuration="7.482750112s" totalImagesPullingTime="4.487158401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:50.48236725 +0000 UTC m=+56.974558926" watchObservedRunningTime="2026-07-02 23:11:50.482750112 +0000 UTC m=+56.974941812" Jul 02 23:11:50.502896 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.502846 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d5shl" podStartSLOduration=2.545752676 podStartE2EDuration="4.502827208s" totalImagesPullingTime="1.957074532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:50.501827363 +0000 UTC m=+56.994019045" watchObservedRunningTime="2026-07-02 23:11:50.502827208 +0000 UTC m=+56.995018883" Jul 02 23:11:50.506989 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.506951 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:50.526412 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:50.526353 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9b49d8864-9gzk7" podStartSLOduration=1.70530129 podStartE2EDuration="5.526336381s" totalImagesPullingTime="3.821035091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:50.525650127 +0000 UTC m=+57.017841802" watchObservedRunningTime="2026-07-02 23:11:50.526336381 +0000 UTC m=+57.018528056" Jul 02 23:11:51.070670 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:51.070635 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:51.165241 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:51.165206 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:51.870710 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:51.870678 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.105142 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.105091 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.195270 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.195182 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw"] Jul 02 23:11:52.198563 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.198526 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.201869 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.201846 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 23:11:52.203229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.203209 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 23:11:52.204769 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.204685 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dzgg6\"" Jul 02 23:11:52.231961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.231928 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw"] Jul 02 23:11:52.248156 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.248121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.248314 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.248188 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.248314 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.248225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrftt\" (UniqueName: \"kubernetes.io/projected/4d75c08d-69a4-4063-a0ee-0af0190b793a-kube-api-access-jrftt\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.248314 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.248267 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d75c08d-69a4-4063-a0ee-0af0190b793a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.267465 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.267428 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw"] Jul 02 23:11:52.269669 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.269646 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.270467 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.270453 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.273907 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.273888 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9ccbx\"" Jul 02 23:11:52.274354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.274334 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 23:11:52.274440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.274375 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 23:11:52.274440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.274386 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 23:11:52.283414 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.283394 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5lncx"] Jul 02 23:11:52.286432 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.286414 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.289647 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.289628 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 23:11:52.290288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.290272 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 23:11:52.290288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.290282 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-82t8x\"" Jul 02 23:11:52.290499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.290481 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw"] Jul 02 23:11:52.290567 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.290552 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 23:11:52.349456 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.349431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrftt\" (UniqueName: \"kubernetes.io/projected/4d75c08d-69a4-4063-a0ee-0af0190b793a-kube-api-access-jrftt\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.349645 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.349465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766e3b40-8634-4674-b049-ecb926555f2e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.349645 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.349487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-metrics-client-ca\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.349645 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.349551 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d75c08d-69a4-4063-a0ee-0af0190b793a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.349780 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.349577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.350101 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350075 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-tls\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350126 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.350197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-sys\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350303 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350287 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.350396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350378 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-textfile\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350520 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350504 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668l2\" (UniqueName: \"kubernetes.io/projected/766e3b40-8634-4674-b049-ecb926555f2e-kube-api-access-668l2\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.350603 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-root\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350603 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350597 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm42m\" (UniqueName: \"kubernetes.io/projected/d75c877b-90a7-4fab-855e-9db2d266ce0f-kube-api-access-hm42m\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350709 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-wtmp\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.350709 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.350826 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350790 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.350893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.350944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350908 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.351013 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.350990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.354342 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:52.351262 2561 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 23:11:52.354342 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:52.351332 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls podName:4d75c08d-69a4-4063-a0ee-0af0190b793a nodeName:}" failed. No retries permitted until 2026-07-02 23:11:52.851313548 +0000 UTC m=+59.343505204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-688kw" (UID: "4d75c08d-69a4-4063-a0ee-0af0190b793a") : secret "openshift-state-metrics-tls" not found Jul 02 23:11:52.354342 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.351587 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d75c08d-69a4-4063-a0ee-0af0190b793a-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.355374 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.355351 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.362194 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.362171 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrftt\" (UniqueName: \"kubernetes.io/projected/4d75c08d-69a4-4063-a0ee-0af0190b793a-kube-api-access-jrftt\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.395777 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.395745 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.452034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.451941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766e3b40-8634-4674-b049-ecb926555f2e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.452034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.451984 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-metrics-client-ca\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.452034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-tls\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452055 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452077 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-sys\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452155 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-sys\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-textfile\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668l2\" (UniqueName: \"kubernetes.io/projected/766e3b40-8634-4674-b049-ecb926555f2e-kube-api-access-668l2\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-root\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452317 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm42m\" (UniqueName: \"kubernetes.io/projected/d75c877b-90a7-4fab-855e-9db2d266ce0f-kube-api-access-hm42m\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-wtmp\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.452429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.453087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452747 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-root\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.452789 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/766e3b40-8634-4674-b049-ecb926555f2e-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.453087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.453029 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-textfile\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453334 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.453129 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-metrics-client-ca\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453334 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.453155 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-wtmp\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453558 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.453479 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-accelerators-collector-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.453925 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.453900 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.454584 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.454556 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/766e3b40-8634-4674-b049-ecb926555f2e-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.455104 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.455078 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-tls\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.455218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.455125 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.455218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.455165 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/766e3b40-8634-4674-b049-ecb926555f2e-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.455367 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.455346 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d75c877b-90a7-4fab-855e-9db2d266ce0f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.467276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.467253 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668l2\" (UniqueName: \"kubernetes.io/projected/766e3b40-8634-4674-b049-ecb926555f2e-kube-api-access-668l2\") pod \"kube-state-metrics-7c9dbb8b7d-nhgpw\" (UID: \"766e3b40-8634-4674-b049-ecb926555f2e\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.469923 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.469900 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm42m\" (UniqueName: \"kubernetes.io/projected/d75c877b-90a7-4fab-855e-9db2d266ce0f-kube-api-access-hm42m\") pod \"node-exporter-5lncx\" (UID: \"d75c877b-90a7-4fab-855e-9db2d266ce0f\") " pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.484336 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.484310 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.577805 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.577776 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:52.579624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.579605 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" Jul 02 23:11:52.595051 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.595019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5lncx" Jul 02 23:11:52.606352 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:52.606325 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75c877b_90a7_4fab_855e_9db2d266ce0f.slice/crio-86e9a42e0427515e33a0dc0933fde81e88f0ee651faf410725811997f69f2f50 WatchSource:0}: Error finding container 86e9a42e0427515e33a0dc0933fde81e88f0ee651faf410725811997f69f2f50: Status 404 returned error can't find the container with id 86e9a42e0427515e33a0dc0933fde81e88f0ee651faf410725811997f69f2f50 Jul 02 23:11:52.719634 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.719557 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw"] Jul 02 23:11:52.721937 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:52.721909 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766e3b40_8634_4674_b049_ecb926555f2e.slice/crio-dc25a6e43aa3b99b374a9cc6215b0e5f5285d91f5e1c9fa5a8832a7c0a620577 WatchSource:0}: Error finding container dc25a6e43aa3b99b374a9cc6215b0e5f5285d91f5e1c9fa5a8832a7c0a620577: Status 404 returned error can't find the container with id dc25a6e43aa3b99b374a9cc6215b0e5f5285d91f5e1c9fa5a8832a7c0a620577 Jul 02 23:11:52.856380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.856348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:52.858851 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:52.858822 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d75c08d-69a4-4063-a0ee-0af0190b793a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-688kw\" (UID: \"4d75c08d-69a4-4063-a0ee-0af0190b793a\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:53.027739 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.027700 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:53.107112 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.107073 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" Jul 02 23:11:53.267006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.266971 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw"] Jul 02 23:11:53.279387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.278675 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:53.407243 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.407209 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 23:11:53.412719 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.412686 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.416489 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.416462 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 23:11:53.417137 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.416792 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 23:11:53.417137 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.416995 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 23:11:53.417411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.417169 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 23:11:53.418371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.417896 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 23:11:53.418371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.418091 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 23:11:53.418371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.418250 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 23:11:53.418371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.418275 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 23:11:53.418672 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.418419 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8zcfp\"" Jul 02 23:11:53.418672 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.418512 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 23:11:53.432160 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.432137 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 23:11:53.443101 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.443072 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:53.452119 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:53.452090 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d75c08d_69a4_4063_a0ee_0af0190b793a.slice/crio-e80acc36d8a285b88383064072134dcd4a6bd15a9566d59259ff61827ea4bff8 WatchSource:0}: Error finding container e80acc36d8a285b88383064072134dcd4a6bd15a9566d59259ff61827ea4bff8: Status 404 returned error can't find the container with id e80acc36d8a285b88383064072134dcd4a6bd15a9566d59259ff61827ea4bff8 Jul 02 23:11:53.460933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.460898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461039 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.460954 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461104 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461037 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461104 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461077 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461105 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461160 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461252 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461283 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461307 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461329 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461360 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzhf\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-kube-api-access-8pzhf\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.461599 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.462039 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.461998 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" event={"ID":"4d75c08d-69a4-4063-a0ee-0af0190b793a","Type":"ContainerStarted","Data":"e80acc36d8a285b88383064072134dcd4a6bd15a9566d59259ff61827ea4bff8"} Jul 02 23:11:53.463457 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.463433 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5lncx" event={"ID":"d75c877b-90a7-4fab-855e-9db2d266ce0f","Type":"ContainerStarted","Data":"86e9a42e0427515e33a0dc0933fde81e88f0ee651faf410725811997f69f2f50"} Jul 02 23:11:53.465060 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.465034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" event={"ID":"766e3b40-8634-4674-b049-ecb926555f2e","Type":"ContainerStarted","Data":"dc25a6e43aa3b99b374a9cc6215b0e5f5285d91f5e1c9fa5a8832a7c0a620577"} Jul 02 23:11:53.561994 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.561964 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562001 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzhf\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-kube-api-access-8pzhf\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562101 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562176 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562202 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562229 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.562429 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:53.562369 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle podName:fa9a8d0b-e257-4acf-8089-f9d5d539ab40 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:54.062346945 +0000 UTC m=+60.554538598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fa9a8d0b-e257-4acf-8089-f9d5d539ab40") : configmap references non-existent config key: ca-bundle.crt Jul 02 23:11:53.562735 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:53.562442 2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 02 23:11:53.562735 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:53.562523 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls podName:fa9a8d0b-e257-4acf-8089-f9d5d539ab40 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:54.062504473 +0000 UTC m=+60.554696126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "fa9a8d0b-e257-4acf-8089-f9d5d539ab40") : secret "alertmanager-main-tls" not found Jul 02 23:11:53.562948 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.562927 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.563967 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.563940 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.570143 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.569974 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.570447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.570281 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.570447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.570380 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.570770 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.570730 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.571746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.571727 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.571949 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.571924 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.572225 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.572197 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.572300 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.572197 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.582334 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.580290 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzhf\" (UniqueName: \"kubernetes.io/projected/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-kube-api-access-8pzhf\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:53.890788 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:53.890692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:54.061426 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.061406 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:54.061746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.061551 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:54.067189 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.067147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:54.067301 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.067193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:54.068388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.068358 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:54.068520 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.068502 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:54.069645 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.069617 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9a8d0b-e257-4acf-8089-f9d5d539ab40-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9a8d0b-e257-4acf-8089-f9d5d539ab40\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:54.329936 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.329881 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8zcfp\"" Jul 02 23:11:54.337463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.337431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 23:11:54.352970 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.352938 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xhdvp" Jul 02 23:11:54.476404 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.476241 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" event={"ID":"4d75c08d-69a4-4063-a0ee-0af0190b793a","Type":"ContainerStarted","Data":"bc661ec438e7d9422997097e58947639a56b20dc9c3a79af0af8b66d00799e85"} Jul 02 23:11:54.476404 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.476354 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" event={"ID":"4d75c08d-69a4-4063-a0ee-0af0190b793a","Type":"ContainerStarted","Data":"7d411279d0802d24807d931f5489eb42a7bd71aa465249842c2bf697c6402377"} Jul 02 23:11:54.478616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.478473 2561 generic.go:356] "Generic (PLEG): container finished" podID="d75c877b-90a7-4fab-855e-9db2d266ce0f" containerID="4b87c07e7c0df18da82654381708bd06ce53b63f7a27d8a43f122f96ebac0e88" exitCode=0 Jul 02 23:11:54.478616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.478583 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5lncx" event={"ID":"d75c877b-90a7-4fab-855e-9db2d266ce0f","Type":"ContainerDied","Data":"4b87c07e7c0df18da82654381708bd06ce53b63f7a27d8a43f122f96ebac0e88"} Jul 02 23:11:54.482391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.482321 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" event={"ID":"766e3b40-8634-4674-b049-ecb926555f2e","Type":"ContainerStarted","Data":"2a70e001696993313e4ddd007488c4c21ea44d46b51f812b6b257a8be30a5265"} Jul 02 23:11:54.482391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.482378 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" event={"ID":"766e3b40-8634-4674-b049-ecb926555f2e","Type":"ContainerStarted","Data":"9dc773a4749b1803998b3e97d14f7200ec14db9d449368d71bf818eb25fd285f"} Jul 02 23:11:54.483090 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.482409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" event={"ID":"766e3b40-8634-4674-b049-ecb926555f2e","Type":"ContainerStarted","Data":"bb47c5acb20b5c971e56e58771e31158f9f01912357afb3bbd12a05022616189"} Jul 02 23:11:54.488670 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.488623 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:11:54.545482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.545252 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 23:11:54.578177 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.577963 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-nhgpw" podStartSLOduration=1.2449433619999999 podStartE2EDuration="2.577942503s" totalImagesPullingTime="1.332999141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:54.576211989 +0000 UTC m=+61.068403675" watchObservedRunningTime="2026-07-02 23:11:54.577942503 +0000 UTC m=+61.070134177" Jul 02 23:11:54.593441 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.593374 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:54.682724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:54.682628 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:54.792746 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:54.792712 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9a8d0b_e257_4acf_8089_f9d5d539ab40.slice/crio-796e198a8a9b93aa2180b9b01994499f0c6577ec1ae08a66aa5464316e841672 WatchSource:0}: Error finding container 796e198a8a9b93aa2180b9b01994499f0c6577ec1ae08a66aa5464316e841672: Status 404 returned error can't find the container with id 796e198a8a9b93aa2180b9b01994499f0c6577ec1ae08a66aa5464316e841672 Jul 02 23:11:55.239578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.239548 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7477f7856-rtvgb"] Jul 02 23:11:55.250999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.250967 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:55.274740 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.274714 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7477f7856-rtvgb"] Jul 02 23:11:55.274866 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.274838 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.279087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.279066 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 02 23:11:55.279380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.279364 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 02 23:11:55.279380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.279370 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2uk44n2lf2ev6\"" Jul 02 23:11:55.279696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.279681 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 02 23:11:55.280223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.280068 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 02 23:11:55.280223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.280130 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 02 23:11:55.280223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.280135 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vgs85\"" Jul 02 23:11:55.333108 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.333076 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:55.384089 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384234 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b547ba94-c72a-456e-9959-9c264027b6fc-metrics-client-ca\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384234 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384132 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4cr\" (UniqueName: \"kubernetes.io/projected/b547ba94-c72a-456e-9959-9c264027b6fc-kube-api-access-wn4cr\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384234 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384363 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384233 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384363 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384298 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384363 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.384363 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.384336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-grpc-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.399886 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.399854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:55.422146 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.422074 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qb68g" Jul 02 23:11:55.445009 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.444980 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:55.445207 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.445034 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:55.454474 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.454182 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.485632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.485740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.485772 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.485808 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-grpc-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.485957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.486000 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b547ba94-c72a-456e-9959-9c264027b6fc-metrics-client-ca\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.486055 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4cr\" (UniqueName: \"kubernetes.io/projected/b547ba94-c72a-456e-9959-9c264027b6fc-kube-api-access-wn4cr\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.486099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.487910 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.487874 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b547ba94-c72a-456e-9959-9c264027b6fc-metrics-client-ca\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.489394 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.489312 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.490074 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.490043 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.491785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.491295 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.491785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.491322 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.491785 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.491719 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-grpc-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.492616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.492578 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b547ba94-c72a-456e-9959-9c264027b6fc-secret-thanos-querier-tls\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.492768 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.492723 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" event={"ID":"4d75c08d-69a4-4063-a0ee-0af0190b793a","Type":"ContainerStarted","Data":"4363c16dec9619034ffec432ad802b51f29b0945891cd0307dac11252cc461dd"} Jul 02 23:11:55.496718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.496623 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4cr\" (UniqueName: \"kubernetes.io/projected/b547ba94-c72a-456e-9959-9c264027b6fc-kube-api-access-wn4cr\") pod \"thanos-querier-7477f7856-rtvgb\" (UID: \"b547ba94-c72a-456e-9959-9c264027b6fc\") " pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.496718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.496675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5lncx" event={"ID":"d75c877b-90a7-4fab-855e-9db2d266ce0f","Type":"ContainerStarted","Data":"f1a34104a25857b6007b2dfc68a2a8e63d958d81d6da5dbcf6594223c41e90d3"} Jul 02 23:11:55.496906 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.496721 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5lncx" event={"ID":"d75c877b-90a7-4fab-855e-9db2d266ce0f","Type":"ContainerStarted","Data":"40e8e052f83c22c9e71b1cebc72c5f7536e6bcd1bfd552a0f1590c9dddada607"} Jul 02 23:11:55.498465 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.498432 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"796e198a8a9b93aa2180b9b01994499f0c6577ec1ae08a66aa5464316e841672"} Jul 02 23:11:55.500438 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.500332 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:55.504032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.504011 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:11:55.516017 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.515968 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-688kw" podStartSLOduration=1.958590696 podStartE2EDuration="3.515949524s" totalImagesPullingTime="1.557358828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:55.514444972 +0000 UTC m=+62.006636649" watchObservedRunningTime="2026-07-02 23:11:55.515949524 +0000 UTC m=+62.008141200" Jul 02 23:11:55.539865 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.539793 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5lncx" podStartSLOduration=2.566813405 podStartE2EDuration="3.539775684s" totalImagesPullingTime="894.273755ms" totalInitContainerRuntime="78.688524ms" isStatefulPod=true podCreationTimestamp="2026-07-02 23:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:55.539014484 +0000 UTC m=+62.031206184" watchObservedRunningTime="2026-07-02 23:11:55.539775684 +0000 UTC m=+62.031967361" Jul 02 23:11:55.585072 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.585035 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:11:55.599828 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.599794 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:11:55.601883 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.601854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:55.763262 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:55.763228 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7477f7856-rtvgb"] Jul 02 23:11:56.034720 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:56.034683 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb547ba94_c72a_456e_9959_9c264027b6fc.slice/crio-b2c48948bb35da0c1cb3664b5a652e9b7553e0d2e6cc1a2c44ff47fc59c51e39 WatchSource:0}: Error finding container b2c48948bb35da0c1cb3664b5a652e9b7553e0d2e6cc1a2c44ff47fc59c51e39: Status 404 returned error can't find the container with id b2c48948bb35da0c1cb3664b5a652e9b7553e0d2e6cc1a2c44ff47fc59c51e39 Jul 02 23:11:56.327839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.327744 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.389888 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.389859 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.504930 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.504859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"b2c48948bb35da0c1cb3664b5a652e9b7553e0d2e6cc1a2c44ff47fc59c51e39"} Jul 02 23:11:56.506644 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.506616 2561 generic.go:356] "Generic (PLEG): container finished" podID="fa9a8d0b-e257-4acf-8089-f9d5d539ab40" containerID="d8139a50b980623b41a295c353f43243c3cf577585b3b6abf24631e4aa82d504" exitCode=0 Jul 02 23:11:56.506800 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.506704 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerDied","Data":"d8139a50b980623b41a295c353f43243c3cf577585b3b6abf24631e4aa82d504"} Jul 02 23:11:56.589917 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.589819 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.629359 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.629319 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f6f5b5756-nmbl7"] Jul 02 23:11:56.632074 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.632052 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.635117 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.634938 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 23:11:56.635223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.635116 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 02 23:11:56.635223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.635185 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 02 23:11:56.635348 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.635236 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dfp18cs5jidq5\"" Jul 02 23:11:56.635348 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.635311 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 02 23:11:56.635606 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.635585 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fg8j9\"" Jul 02 23:11:56.643286 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.643257 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f6f5b5756-nmbl7"] Jul 02 23:11:56.663355 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.663321 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.700834 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.700791 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhq4\" (UniqueName: \"kubernetes.io/projected/6c8bac24-890b-4b15-95ae-39d365f41490-kube-api-access-kdhq4\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701012 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.700838 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-client-certs\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701012 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.700943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-tls\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701012 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.700981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701012 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.701002 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-metrics-server-audit-profiles\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701174 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.701063 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c8bac24-890b-4b15-95ae-39d365f41490-audit-log\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.701174 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.701125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-client-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.731923 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.731891 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.802255 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802139 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-client-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802255 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhq4\" (UniqueName: \"kubernetes.io/projected/6c8bac24-890b-4b15-95ae-39d365f41490-kube-api-access-kdhq4\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802255 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-client-certs\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-tls\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802385 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-metrics-server-audit-profiles\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.802623 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.802416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c8bac24-890b-4b15-95ae-39d365f41490-audit-log\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.803037 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.803008 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c8bac24-890b-4b15-95ae-39d365f41490-audit-log\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.803318 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.803256 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.803614 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.803588 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c8bac24-890b-4b15-95ae-39d365f41490-metrics-server-audit-profiles\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.805216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.805193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-client-certs\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.805519 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.805499 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-client-ca-bundle\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.805519 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.805507 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c8bac24-890b-4b15-95ae-39d365f41490-secret-metrics-server-tls\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.811692 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.811664 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhq4\" (UniqueName: \"kubernetes.io/projected/6c8bac24-890b-4b15-95ae-39d365f41490-kube-api-access-kdhq4\") pod \"metrics-server-f6f5b5756-nmbl7\" (UID: \"6c8bac24-890b-4b15-95ae-39d365f41490\") " pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:56.903354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.903279 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:56.907160 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.907126 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9"] Jul 02 23:11:56.910076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.910059 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:56.912727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.912697 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 02 23:11:56.912727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.912718 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n4gn7\"" Jul 02 23:11:56.921880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.921856 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9"] Jul 02 23:11:56.944229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:56.944205 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:11:57.002686 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.002649 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:57.003720 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.003691 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.003844 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.003809 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wpmg\" (UniqueName: \"kubernetes.io/projected/867ab172-93ec-4123-a022-0037bdc18c40-kube-api-access-5wpmg\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.104737 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.104708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.104904 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.104828 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wpmg\" (UniqueName: \"kubernetes.io/projected/867ab172-93ec-4123-a022-0037bdc18c40-kube-api-access-5wpmg\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.104978 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:57.104896 2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 02 23:11:57.105032 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:11:57.104989 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert podName:867ab172-93ec-4123-a022-0037bdc18c40 nodeName:}" failed. No retries permitted until 2026-07-02 23:11:57.60496515 +0000 UTC m=+64.097156819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-kbbh9" (UID: "867ab172-93ec-4123-a022-0037bdc18c40") : secret "monitoring-plugin-cert" not found Jul 02 23:11:57.116305 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.116250 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wpmg\" (UniqueName: \"kubernetes.io/projected/867ab172-93ec-4123-a022-0037bdc18c40-kube-api-access-5wpmg\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.118137 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.118109 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f6f5b5756-nmbl7"] Jul 02 23:11:57.121455 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:57.121432 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8bac24_890b_4b15_95ae_39d365f41490.slice/crio-6e57a1ae51fca05698bcac3779ec3ddcd9ee7f49a12b431f77b6a5ddc75e707f WatchSource:0}: Error finding container 6e57a1ae51fca05698bcac3779ec3ddcd9ee7f49a12b431f77b6a5ddc75e707f: Status 404 returned error can't find the container with id 6e57a1ae51fca05698bcac3779ec3ddcd9ee7f49a12b431f77b6a5ddc75e707f Jul 02 23:11:57.386041 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.386005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:57.513904 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.513821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" event={"ID":"6c8bac24-890b-4b15-95ae-39d365f41490","Type":"ContainerStarted","Data":"6e57a1ae51fca05698bcac3779ec3ddcd9ee7f49a12b431f77b6a5ddc75e707f"} Jul 02 23:11:57.609791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.609756 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.612521 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.612496 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/867ab172-93ec-4123-a022-0037bdc18c40-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-kbbh9\" (UID: \"867ab172-93ec-4123-a022-0037bdc18c40\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:57.823152 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:57.823082 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:11:58.208288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.208118 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9"] Jul 02 23:11:58.211161 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:58.211138 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867ab172_93ec_4123_a022_0037bdc18c40.slice/crio-ea8a1e240df2d29d607af33a3b903627a35b91b8219f4a311dc790ddf3b7ccb1 WatchSource:0}: Error finding container ea8a1e240df2d29d607af33a3b903627a35b91b8219f4a311dc790ddf3b7ccb1: Status 404 returned error can't find the container with id ea8a1e240df2d29d607af33a3b903627a35b91b8219f4a311dc790ddf3b7ccb1 Jul 02 23:11:58.296718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.296585 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:11:58.300492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.300067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.312921 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.312868 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:11:58.416602 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.416564 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmblj\" (UniqueName: \"kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.416632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.416734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.416953 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.416995 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.417061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.417218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.417111 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.520832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.520902 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmblj\" (UniqueName: \"kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.520955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.521034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.521144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.521195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.521272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.522160 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.522391 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.522945 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.524448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.523041 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.526179 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.526095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.528174 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.528142 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.531446 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.531411 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"c01b36c3cf3194a6e21d35c7aeeb4b6c3ac66bf023db01c0770d790fc1e7bd67"} Jul 02 23:11:58.531624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.531526 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"1d722dfa0ce50c386fa14b793d124262bcf1490ecf22925ba8f48b23bd1fc97c"} Jul 02 23:11:58.531624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.531587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"0c81cefbab6966eed8031ce86abce844d06f1dc28ffc1629d0760943c3bcde15"} Jul 02 23:11:58.533836 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.533675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" event={"ID":"867ab172-93ec-4123-a022-0037bdc18c40","Type":"ContainerStarted","Data":"ea8a1e240df2d29d607af33a3b903627a35b91b8219f4a311dc790ddf3b7ccb1"} Jul 02 23:11:58.539433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.539399 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmblj\" (UniqueName: \"kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj\") pod \"console-6bbd88b6f6-k9d9n\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.541088 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.540693 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"ccfeb782e712d9d6cade5dae96ce123895d1e13d036a23f9e740139a6dd452e3"} Jul 02 23:11:58.541088 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.540756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"7a33131bb0f53f0441b5df649b16a3478e184b7aae83d66fe5e42090fed96295"} Jul 02 23:11:58.541088 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.540782 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"c0928337b126da6550c02ef9d9298c37560e9a73e176daad6bc4c6ff96f0e2da"} Jul 02 23:11:58.541088 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.540804 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"b9b004d8c149b06caea202007a8f386cb6eb98d4f79e340c9ef0e2f863cc0388"} Jul 02 23:11:58.541088 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.540827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"86841c3f36c08afc337d53fcd65c499ad9cc52a31539870856c42137d733f9a0"} Jul 02 23:11:58.567357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.567321 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:58.575766 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.575735 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 23:11:58.579756 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.579682 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.583562 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.583512 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 23:11:58.583727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.583700 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2jas0ajmh39eb\"" Jul 02 23:11:58.585696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.585657 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 23:11:58.585849 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.585834 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 23:11:58.586058 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.586041 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 23:11:58.587408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.587164 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 23:11:58.587408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.587186 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 23:11:58.587408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.587223 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 23:11:58.587408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.587150 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 23:11:58.588820 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.588500 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 23:11:58.590060 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.589385 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q9hb7\"" Jul 02 23:11:58.590396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.590072 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 23:11:58.590396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.590243 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 23:11:58.590396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.590255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 23:11:58.590396 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.590389 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 23:11:58.591306 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.591247 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 23:11:58.608648 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.608614 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 23:11:58.620148 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.620118 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622555 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622611 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622667 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxcl\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-kube-api-access-fkxcl\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622765 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.622783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622789 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622850 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.622964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623042 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623077 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623101 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623129 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.623633 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.623248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.700457 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.700363 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:11:58.724436 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724526 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724654 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724712 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.724791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724831 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724861 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724952 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.724985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxcl\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-kube-api-access-fkxcl\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.725025 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725406 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.725054 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.725406 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.725084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.726859 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.725803 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.726859 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.726089 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.727721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.727695 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.728336 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.728676 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.729113 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.729301 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.729612 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.729955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.729775 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79c921ba-7a0d-4e2d-be19-7ee98042d786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.733660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.733613 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-web-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.735983 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.734911 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.735983 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.734986 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.735983 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.735449 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.735983 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.735829 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-config\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.736299 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.736245 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.736472 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.736447 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79c921ba-7a0d-4e2d-be19-7ee98042d786-config-out\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.737224 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.737203 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.737730 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.737689 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.739626 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.739587 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79c921ba-7a0d-4e2d-be19-7ee98042d786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.742723 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.742676 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxcl\" (UniqueName: \"kubernetes.io/projected/79c921ba-7a0d-4e2d-be19-7ee98042d786-kube-api-access-fkxcl\") pod \"prometheus-k8s-0\" (UID: \"79c921ba-7a0d-4e2d-be19-7ee98042d786\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.806354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.806270 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:11:58.812234 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:58.812187 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a8dc3b_4b24_4f64_9440_268ce87086f6.slice/crio-52caba4c5570a80e7422bada41ac61063a5e146657b29818582fb5ffc9da1f4b WatchSource:0}: Error finding container 52caba4c5570a80e7422bada41ac61063a5e146657b29818582fb5ffc9da1f4b: Status 404 returned error can't find the container with id 52caba4c5570a80e7422bada41ac61063a5e146657b29818582fb5ffc9da1f4b Jul 02 23:11:58.827029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.826318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:58.827029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.826386 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:58.830466 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.829728 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 23:11:58.830466 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.830100 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 23:11:58.841354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.841290 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a7cc66b4-608b-4631-827a-2c764e4e023a-original-pull-secret\") pod \"global-pull-secret-syncer-8tm84\" (UID: \"a7cc66b4-608b-4631-827a-2c764e4e023a\") " pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:58.845150 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.845105 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f51cd4b-2792-414b-b9c0-54dcb8e390c0-metrics-certs\") pod \"network-metrics-daemon-hf4vd\" (UID: \"1f51cd4b-2792-414b-b9c0-54dcb8e390c0\") " pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:58.879789 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.879570 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prtl8\"" Jul 02 23:11:58.886131 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.885679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tm84" Jul 02 23:11:58.887340 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.887267 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hf4vd" Jul 02 23:11:58.898073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.897527 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:11:58.929341 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.928152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:58.931452 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.931212 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 23:11:58.941796 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.941558 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 23:11:58.956660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:58.956598 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrxr\" (UniqueName: \"kubernetes.io/projected/06152e08-c5a4-4607-92df-af45954f43ba-kube-api-access-bwrxr\") pod \"network-check-target-c2pwk\" (UID: \"06152e08-c5a4-4607-92df-af45954f43ba\") " pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:59.118219 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.118081 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tm84"] Jul 02 23:11:59.170939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.169143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 23:11:59.184029 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.184001 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmk5n\"" Jul 02 23:11:59.187067 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.187044 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hf4vd"] Jul 02 23:11:59.190842 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.190817 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:11:59.547510 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.547474 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbd88b6f6-k9d9n" event={"ID":"b3a8dc3b-4b24-4f64-9440-268ce87086f6","Type":"ContainerStarted","Data":"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499"} Jul 02 23:11:59.548030 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.547546 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbd88b6f6-k9d9n" event={"ID":"b3a8dc3b-4b24-4f64-9440-268ce87086f6","Type":"ContainerStarted","Data":"52caba4c5570a80e7422bada41ac61063a5e146657b29818582fb5ffc9da1f4b"} Jul 02 23:11:59.567723 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:11:59.567656 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bbd88b6f6-k9d9n" podStartSLOduration=1.5676317210000001 podStartE2EDuration="1.567631721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:11:59.566763623 +0000 UTC m=+66.058955306" watchObservedRunningTime="2026-07-02 23:11:59.567631721 +0000 UTC m=+66.059823397" Jul 02 23:11:59.633645 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:59.633608 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f51cd4b_2792_414b_b9c0_54dcb8e390c0.slice/crio-50350b7bb431f6c8859dc4a331895d220915913094ac608d5cd72092c6d8af4e WatchSource:0}: Error finding container 50350b7bb431f6c8859dc4a331895d220915913094ac608d5cd72092c6d8af4e: Status 404 returned error can't find the container with id 50350b7bb431f6c8859dc4a331895d220915913094ac608d5cd72092c6d8af4e Jul 02 23:11:59.960230 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:59.960185 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cc66b4_608b_4631_827a_2c764e4e023a.slice/crio-f9bc7de1aace437ee3b7c8706ec409eddd6d8a54ab93683f2f9c9552293b174d WatchSource:0}: Error finding container f9bc7de1aace437ee3b7c8706ec409eddd6d8a54ab93683f2f9c9552293b174d: Status 404 returned error can't find the container with id f9bc7de1aace437ee3b7c8706ec409eddd6d8a54ab93683f2f9c9552293b174d Jul 02 23:11:59.961377 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:11:59.961349 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c921ba_7a0d_4e2d_be19_7ee98042d786.slice/crio-ac26803248efb3c1308a617069b4a5c29cb798f89fb95b0a18ec13847fd78ce7 WatchSource:0}: Error finding container ac26803248efb3c1308a617069b4a5c29cb798f89fb95b0a18ec13847fd78ce7: Status 404 returned error can't find the container with id ac26803248efb3c1308a617069b4a5c29cb798f89fb95b0a18ec13847fd78ce7 Jul 02 23:12:00.238159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.238079 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c2pwk"] Jul 02 23:12:00.284983 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:12:00.284945 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06152e08_c5a4_4607_92df_af45954f43ba.slice/crio-4b2f9bcd8c50e382fc04f8ef3cbe84f5f4beeea145430ee4c3a8e542ae197431 WatchSource:0}: Error finding container 4b2f9bcd8c50e382fc04f8ef3cbe84f5f4beeea145430ee4c3a8e542ae197431: Status 404 returned error can't find the container with id 4b2f9bcd8c50e382fc04f8ef3cbe84f5f4beeea145430ee4c3a8e542ae197431 Jul 02 23:12:00.556036 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.555991 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" event={"ID":"6c8bac24-890b-4b15-95ae-39d365f41490","Type":"ContainerStarted","Data":"633ae7e2132d2ac5f9fb4e880838d1709a279ee7200141889e2636cb81705a02"} Jul 02 23:12:00.561394 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.561358 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"c4b2969e4af4e707c3c6de3844e38a4db537b3b6d8c6460e4f1e2f149e9a5f37"} Jul 02 23:12:00.561596 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.561444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"4e7eaeb58cd40b42dfb00bcfbb6221820133bbd6abd30cf118f99f38cb80563a"} Jul 02 23:12:00.561596 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.561493 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" event={"ID":"b547ba94-c72a-456e-9959-9c264027b6fc","Type":"ContainerStarted","Data":"bb5b95a939b1c3ef09ca793a4fbc48384d92f28f3dfe1576122b14e6609d5503"} Jul 02 23:12:00.561596 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.561555 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:12:00.563648 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.563615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" event={"ID":"867ab172-93ec-4123-a022-0037bdc18c40","Type":"ContainerStarted","Data":"f156b12b1bb409f0747063c3bf2d74b58229d401fd3b79cc01fac4236d147e74"} Jul 02 23:12:00.564127 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.564100 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:12:00.566327 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.566301 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c2pwk" event={"ID":"06152e08-c5a4-4607-92df-af45954f43ba","Type":"ContainerStarted","Data":"4b2f9bcd8c50e382fc04f8ef3cbe84f5f4beeea145430ee4c3a8e542ae197431"} Jul 02 23:12:00.568467 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.568443 2561 generic.go:356] "Generic (PLEG): container finished" podID="79c921ba-7a0d-4e2d-be19-7ee98042d786" containerID="47be99e6aa5849f03c76164ae8c1532b85e641a6fb26690e790a22750c74c8fd" exitCode=0 Jul 02 23:12:00.568587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.568523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerDied","Data":"47be99e6aa5849f03c76164ae8c1532b85e641a6fb26690e790a22750c74c8fd"} Jul 02 23:12:00.568660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.568592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"ac26803248efb3c1308a617069b4a5c29cb798f89fb95b0a18ec13847fd78ce7"} Jul 02 23:12:00.571344 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.571316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tm84" event={"ID":"a7cc66b4-608b-4631-827a-2c764e4e023a","Type":"ContainerStarted","Data":"f9bc7de1aace437ee3b7c8706ec409eddd6d8a54ab93683f2f9c9552293b174d"} Jul 02 23:12:00.571433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.571396 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" Jul 02 23:12:00.574477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.574438 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hf4vd" event={"ID":"1f51cd4b-2792-414b-b9c0-54dcb8e390c0","Type":"ContainerStarted","Data":"50350b7bb431f6c8859dc4a331895d220915913094ac608d5cd72092c6d8af4e"} Jul 02 23:12:00.576601 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.575333 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" podStartSLOduration=1.6349711409999999 podStartE2EDuration="4.575315552s" totalImagesPullingTime="2.940344411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:00.5741181 +0000 UTC m=+67.066309779" watchObservedRunningTime="2026-07-02 23:12:00.575315552 +0000 UTC m=+67.067507229" Jul 02 23:12:00.579836 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.579791 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9a8d0b-e257-4acf-8089-f9d5d539ab40","Type":"ContainerStarted","Data":"47520fa6dd81de93c1a210926a1927a1f898a26cb9284233be01e6780e96b01d"} Jul 02 23:12:00.629242 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.629169 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" podStartSLOduration=1.824201261 podStartE2EDuration="5.629145499s" totalImagesPullingTime="3.804944238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:00.628086727 +0000 UTC m=+67.120278403" watchObservedRunningTime="2026-07-02 23:12:00.629145499 +0000 UTC m=+67.121337238" Jul 02 23:12:00.644657 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.644588 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-kbbh9" podStartSLOduration=2.794042686 podStartE2EDuration="4.644563446s" totalImagesPullingTime="1.85052076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:00.643909004 +0000 UTC m=+67.136100682" watchObservedRunningTime="2026-07-02 23:12:00.644563446 +0000 UTC m=+67.136755124" Jul 02 23:12:00.676794 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:00.676305 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.208535247 podStartE2EDuration="7.676280998s" totalImagesPullingTime="4.450817982s" totalInitContainerRuntime="16.927769ms" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:00.672796478 +0000 UTC m=+67.164988166" watchObservedRunningTime="2026-07-02 23:12:00.676280998 +0000 UTC m=+67.168472673" Jul 02 23:12:01.589366 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.589314 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hf4vd" event={"ID":"1f51cd4b-2792-414b-b9c0-54dcb8e390c0","Type":"ContainerStarted","Data":"e21ff11cfc9c8460a70f291dbd1f1f8b5518d1e1f21c74361ba20eb0f516accc"} Jul 02 23:12:01.589966 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.589415 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hf4vd" event={"ID":"1f51cd4b-2792-414b-b9c0-54dcb8e390c0","Type":"ContainerStarted","Data":"1032408785b8930334943073601d554e9c565713f33ffb81504e5319f239acb7"} Jul 02 23:12:01.610118 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.610052 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hf4vd" podStartSLOduration=66.475071493 podStartE2EDuration="1m7.610029848s" totalImagesPullingTime="1.134958355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:01.608455018 +0000 UTC m=+68.100646697" watchObservedRunningTime="2026-07-02 23:12:01.610029848 +0000 UTC m=+68.102221524" Jul 02 23:12:01.713055 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.713013 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:01.814906 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.814874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:01.917222 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:01.917130 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:04.015637 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:04.015594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:04.120145 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:04.120049 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:04.949884 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:04.949835 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:05.003485 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:05.003453 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:06.058815 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.058786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:06.130236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.130163 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:06.598492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.598319 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7477f7856-rtvgb" Jul 02 23:12:06.618287 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.618245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c2pwk" event={"ID":"06152e08-c5a4-4607-92df-af45954f43ba","Type":"ContainerStarted","Data":"34314fd17d5fb0cf7b14041fa718c4c26c19a68ce7afa28b18832a784bbce1d7"} Jul 02 23:12:06.618448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.618347 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:12:06.621247 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.621209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"49fce0020faae4d04f0e55349df8a780b7aae53d381ef4ec3144ffb963c9e50d"} Jul 02 23:12:06.622996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.622964 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tm84" event={"ID":"a7cc66b4-608b-4631-827a-2c764e4e023a","Type":"ContainerStarted","Data":"bb815737948f4226a1a73b556ae181074c4b8e96a0ec91309f06bd6fd5ea5935"} Jul 02 23:12:06.673488 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.673422 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8tm84" podStartSLOduration=66.226188552 podStartE2EDuration="1m12.673400218s" totalImagesPullingTime="6.447211666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:06.649925572 +0000 UTC m=+73.142117247" watchObservedRunningTime="2026-07-02 23:12:06.673400218 +0000 UTC m=+73.165591897" Jul 02 23:12:06.674484 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:06.674453 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c2pwk" podStartSLOduration=66.49330391 podStartE2EDuration="1m12.674443015s" totalImagesPullingTime="6.181139105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:06.672296142 +0000 UTC m=+73.164487829" watchObservedRunningTime="2026-07-02 23:12:06.674443015 +0000 UTC m=+73.166634690" Jul 02 23:12:07.632541 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.632500 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"86f015e1fa42a599a0faa2de7b9a72d610be61cdd37d3822f5ad94eda6087177"} Jul 02 23:12:07.632956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.632623 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"703fff632f0f74a280e2d9eaf83588f20badf85a74d0ef5b37359f512e557eb7"} Jul 02 23:12:07.632956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.632661 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"51ceba53a2434f145ca0f7d121328838fda4ff93f9100949dff093f0f9d1440e"} Jul 02 23:12:07.632956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.632693 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"c87032d2f5dcf5e53b44bedc6c5540a3176929716def2f65cd9f92ac534a8bdd"} Jul 02 23:12:07.632956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.632732 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79c921ba-7a0d-4e2d-be19-7ee98042d786","Type":"ContainerStarted","Data":"e08f6b1d4931c4d49f1165c765d6ac6ceea450da68d1ee55adbe288a797d3c40"} Jul 02 23:12:07.665746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:07.665673 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.727714332 podStartE2EDuration="9.665649383s" totalImagesPullingTime="5.907748338s" totalInitContainerRuntime="30.186713ms" isStatefulPod=false podCreationTimestamp="2026-07-02 23:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:12:07.66484491 +0000 UTC m=+74.157036589" watchObservedRunningTime="2026-07-02 23:12:07.665649383 +0000 UTC m=+74.157841059" Jul 02 23:12:08.138291 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.138257 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:08.199058 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.199022 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:08.620786 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.620743 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:12:08.620962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.620810 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:12:08.625696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.625672 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:12:08.640964 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.640937 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:12:08.692688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.692650 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:12:08.807032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.807003 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:08.898646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.898569 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:12:08.905996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.905967 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:08.959990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:08.959960 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:13.009911 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:13.009869 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:16.945124 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:16.945001 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:12:16.945124 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:16.945077 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:12:21.043987 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:21.043950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:21.144689 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:21.144637 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:22.538469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.538431 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-58d697c4b-mr79q" podUID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" containerName="console" containerID="cri-o://ba875863860d7a352d711ad0738b45241beecfac4e88edc0b5581b9e602ede03" gracePeriod=15 Jul 02 23:12:22.702449 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.702415 2561 generic.go:356] "Generic (PLEG): container finished" podID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" containerID="ba875863860d7a352d711ad0738b45241beecfac4e88edc0b5581b9e602ede03" exitCode=0 Jul 02 23:12:22.702622 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.702486 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d697c4b-mr79q" event={"ID":"d95d679f-356a-46ee-a1ff-abab7f1b9f51","Type":"ContainerDied","Data":"ba875863860d7a352d711ad0738b45241beecfac4e88edc0b5581b9e602ede03"} Jul 02 23:12:22.798654 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.797300 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:12:22.972843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.972795 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.972843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.972853 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.973099 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.972907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.973099 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.972942 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.973099 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.973014 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.973099 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.973038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65dbx\" (UniqueName: \"kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx\") pod \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\" (UID: \"d95d679f-356a-46ee-a1ff-abab7f1b9f51\") " Jul 02 23:12:22.973370 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.973337 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:22.973431 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.973369 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca" (OuterVolumeSpecName: "service-ca") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:22.973431 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.973349 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config" (OuterVolumeSpecName: "console-config") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:22.975353 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.975319 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:12:22.975353 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.975325 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx" (OuterVolumeSpecName: "kube-api-access-65dbx") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "kube-api-access-65dbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:12:22.975511 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:22.975422 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d95d679f-356a-46ee-a1ff-abab7f1b9f51" (UID: "d95d679f-356a-46ee-a1ff-abab7f1b9f51"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:12:23.074607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074500 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-oauth-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.074607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074559 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.074607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074574 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-oauth-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.074607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074585 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d95d679f-356a-46ee-a1ff-abab7f1b9f51-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.074607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074598 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95d679f-356a-46ee-a1ff-abab7f1b9f51-console-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.074911 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.074610 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dbx\" (UniqueName: \"kubernetes.io/projected/d95d679f-356a-46ee-a1ff-abab7f1b9f51-kube-api-access-65dbx\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:23.708600 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.708574 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d697c4b-mr79q" Jul 02 23:12:23.708600 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.708585 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d697c4b-mr79q" event={"ID":"d95d679f-356a-46ee-a1ff-abab7f1b9f51","Type":"ContainerDied","Data":"4ea467b39c3e76fc77231e09a591fe90dcb8b475d2e45f670d77ba2852bf4309"} Jul 02 23:12:23.709083 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.708644 2561 scope.go:117] "RemoveContainer" containerID="ba875863860d7a352d711ad0738b45241beecfac4e88edc0b5581b9e602ede03" Jul 02 23:12:23.731221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.731185 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:12:23.735438 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:23.735408 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d697c4b-mr79q"] Jul 02 23:12:24.067354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:24.067310 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" path="/var/lib/kubelet/pods/d95d679f-356a-46ee-a1ff-abab7f1b9f51/volumes" Jul 02 23:12:29.221253 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:29.221210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:33.713191 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:33.713127 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9b49d8864-9gzk7" podUID="233b1304-733c-410d-acce-498fe0a7fd6e" containerName="console" containerID="cri-o://32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229" gracePeriod=15 Jul 02 23:12:33.956025 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:33.956002 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:12:34.066005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.065975 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066019 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066045 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066147 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6btkj\" (UniqueName: \"kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066198 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066421 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066242 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066421 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066293 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config\") pod \"233b1304-733c-410d-acce-498fe0a7fd6e\" (UID: \"233b1304-733c-410d-acce-498fe0a7fd6e\") " Jul 02 23:12:34.066548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066505 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:34.066604 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066564 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:34.066656 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066630 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:34.066846 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.066821 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config" (OuterVolumeSpecName: "console-config") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:12:34.068582 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.068552 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj" (OuterVolumeSpecName: "kube-api-access-6btkj") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "kube-api-access-6btkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:12:34.068695 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.068597 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:12:34.068860 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.068830 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "233b1304-733c-410d-acce-498fe0a7fd6e" (UID: "233b1304-733c-410d-acce-498fe0a7fd6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:12:34.133721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.133681 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:34.166889 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166859 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6btkj\" (UniqueName: \"kubernetes.io/projected/233b1304-733c-410d-acce-498fe0a7fd6e-kube-api-access-6btkj\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.166889 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166885 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.166889 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166897 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.167080 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166906 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-console-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.167080 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166915 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-oauth-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.167080 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166923 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/233b1304-733c-410d-acce-498fe0a7fd6e-trusted-ca-bundle\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.167080 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.166931 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/233b1304-733c-410d-acce-498fe0a7fd6e-console-oauth-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:12:34.769000 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.768965 2561 generic.go:356] "Generic (PLEG): container finished" podID="233b1304-733c-410d-acce-498fe0a7fd6e" containerID="32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229" exitCode=0 Jul 02 23:12:34.769433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.769009 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b49d8864-9gzk7" Jul 02 23:12:34.769433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.769046 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b49d8864-9gzk7" event={"ID":"233b1304-733c-410d-acce-498fe0a7fd6e","Type":"ContainerDied","Data":"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229"} Jul 02 23:12:34.769433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.769092 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b49d8864-9gzk7" event={"ID":"233b1304-733c-410d-acce-498fe0a7fd6e","Type":"ContainerDied","Data":"7a1bf7bda3c390d6fd4767d6cc1f55323baac85f89f32cb25f78f53fddcfdb4c"} Jul 02 23:12:34.769433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.769118 2561 scope.go:117] "RemoveContainer" containerID="32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229" Jul 02 23:12:34.781055 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.780783 2561 scope.go:117] "RemoveContainer" containerID="32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229" Jul 02 23:12:34.781463 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:12:34.781349 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229\": container with ID starting with 32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229 not found: ID does not exist" containerID="32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229" Jul 02 23:12:34.781463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.781387 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229"} err="failed to get container status \"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229\": rpc error: code = NotFound desc = could not find container \"32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229\": container with ID starting with 32c34c04143b216d563b31872ce43272ce8b4d406401b38bd38f52a51545f229 not found: ID does not exist" Jul 02 23:12:34.791063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.791005 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:12:34.794087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:34.794022 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b49d8864-9gzk7"] Jul 02 23:12:36.045711 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.045672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:36.067787 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.067752 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233b1304-733c-410d-acce-498fe0a7fd6e" path="/var/lib/kubelet/pods/233b1304-733c-410d-acce-498fe0a7fd6e/volumes" Jul 02 23:12:36.128207 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.128171 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:36.483257 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.483147 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-szfsv_f9526752-62a6-4815-8d5e-0a523d822544/serve-healthcheck-canary/0.log" Jul 02 23:12:36.950558 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.950509 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:12:36.954483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:36.954456 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f6f5b5756-nmbl7" Jul 02 23:12:37.635155 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:37.635124 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c2pwk" Jul 02 23:12:44.182822 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:44.182788 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:44.285211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:44.285180 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:44.350270 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:44.350243 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:12:58.898843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:58.898800 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:12:58.918185 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:58.918162 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:12:59.905384 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:12:59.905357 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 23:13:11.532792 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:11.532750 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:11.771927 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:11.771886 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:16.631801 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.631758 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c6f7795bd-7kz8m"] Jul 02 23:13:16.632373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632281 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" containerName="console" Jul 02 23:13:16.632373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632301 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" containerName="console" Jul 02 23:13:16.632373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632327 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233b1304-733c-410d-acce-498fe0a7fd6e" containerName="console" Jul 02 23:13:16.632373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632335 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="233b1304-733c-410d-acce-498fe0a7fd6e" containerName="console" Jul 02 23:13:16.632612 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632414 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="233b1304-733c-410d-acce-498fe0a7fd6e" containerName="console" Jul 02 23:13:16.632612 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.632429 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95d679f-356a-46ee-a1ff-abab7f1b9f51" containerName="console" Jul 02 23:13:16.634857 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.634833 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.638211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.638176 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 02 23:13:16.638344 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.638231 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 02 23:13:16.638586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.638565 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 02 23:13:16.638681 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.638597 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 02 23:13:16.638741 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.638722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t8p78\"" Jul 02 23:13:16.640391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.640370 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 02 23:13:16.644825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.644802 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 02 23:13:16.646403 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.646377 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c6f7795bd-7kz8m"] Jul 02 23:13:16.649750 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.649722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.649859 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.649775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.649929 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.649849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-metrics-client-ca\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.649929 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.649913 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.650033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.649960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.650033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.650002 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-serving-certs-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.650130 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.650030 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm8t\" (UniqueName: \"kubernetes.io/projected/210fa60d-c919-43b4-9ad0-112bb981f4c7-kube-api-access-srm8t\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.650130 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.650087 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-federate-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.750838 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750801 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.750838 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750846 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750870 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-metrics-client-ca\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750938 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750970 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-serving-certs-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.750994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm8t\" (UniqueName: \"kubernetes.io/projected/210fa60d-c919-43b4-9ad0-112bb981f4c7-kube-api-access-srm8t\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751283 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.751253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-federate-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.751812 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-metrics-client-ca\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.751979 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.751890 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.752051 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.752029 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/210fa60d-c919-43b4-9ad0-112bb981f4c7-serving-certs-ca-bundle\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.753616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.753585 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.753761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.753745 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-federate-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.753828 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.753758 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-telemeter-client-tls\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.753828 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.753786 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/210fa60d-c919-43b4-9ad0-112bb981f4c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.760094 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.760072 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm8t\" (UniqueName: \"kubernetes.io/projected/210fa60d-c919-43b4-9ad0-112bb981f4c7-kube-api-access-srm8t\") pod \"telemeter-client-c6f7795bd-7kz8m\" (UID: \"210fa60d-c919-43b4-9ad0-112bb981f4c7\") " pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:16.948825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:16.948749 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" Jul 02 23:13:17.093891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:17.093849 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c6f7795bd-7kz8m"] Jul 02 23:13:17.097070 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:13:17.097038 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210fa60d_c919_43b4_9ad0_112bb981f4c7.slice/crio-0d314c835ead4ba5c2d275d1912975bd28c759899cca82fefe8a378f88bf243c WatchSource:0}: Error finding container 0d314c835ead4ba5c2d275d1912975bd28c759899cca82fefe8a378f88bf243c: Status 404 returned error can't find the container with id 0d314c835ead4ba5c2d275d1912975bd28c759899cca82fefe8a378f88bf243c Jul 02 23:13:17.963763 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:17.963717 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" event={"ID":"210fa60d-c919-43b4-9ad0-112bb981f4c7","Type":"ContainerStarted","Data":"0d314c835ead4ba5c2d275d1912975bd28c759899cca82fefe8a378f88bf243c"} Jul 02 23:13:18.970981 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:18.970946 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" event={"ID":"210fa60d-c919-43b4-9ad0-112bb981f4c7","Type":"ContainerStarted","Data":"8998bb80568fbfa7f0e63833f5a7f14bb76626a1945978a1140bed50ca2967d3"} Jul 02 23:13:19.977216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:19.977172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" event={"ID":"210fa60d-c919-43b4-9ad0-112bb981f4c7","Type":"ContainerStarted","Data":"af279742d9d1670bad0dc1cd5383fab1b22f8cc5359e2fc2fd7ecc1e9917c570"} Jul 02 23:13:19.977663 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:19.977254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" event={"ID":"210fa60d-c919-43b4-9ad0-112bb981f4c7","Type":"ContainerStarted","Data":"2190fd33e703880d1c31e021297202606a478fbf93839aa7fec3643ec3a9a2fc"} Jul 02 23:13:20.003825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:20.003765 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c6f7795bd-7kz8m" podStartSLOduration=2.21355354 podStartE2EDuration="4.003746108s" totalImagesPullingTime="1.790192568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:13:20.002999407 +0000 UTC m=+146.495191110" watchObservedRunningTime="2026-07-02 23:13:20.003746108 +0000 UTC m=+146.495937782" Jul 02 23:13:21.402931 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.402897 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:21.405382 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.405363 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.425165 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.425129 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:21.473472 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.473437 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:21.494716 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.494716 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494720 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.494903 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494758 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.494903 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.494903 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.494903 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494883 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmfh\" (UniqueName: \"kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.495027 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.494935 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.575843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.575816 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:21.595443 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595407 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595443 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmfh\" (UniqueName: \"kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595561 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595619 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595649 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.595864 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.595712 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.596351 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.596321 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.596482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.596404 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.596482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.596424 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.596482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.596466 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.597981 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.597960 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.598053 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.598040 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.604636 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.604616 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmfh\" (UniqueName: \"kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh\") pod \"console-77544c57b7-snjdl\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.675998 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.675920 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:21.714367 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.714329 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:21.852215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.852100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:21.855025 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:13:21.854992 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099ca819_e484_4e5b_a170_d108b401f9e8.slice/crio-a6ac5bc08e73d8cf42939ecb4f43b9bce3f0028d96536cd51f4e891bc7841b3a WatchSource:0}: Error finding container a6ac5bc08e73d8cf42939ecb4f43b9bce3f0028d96536cd51f4e891bc7841b3a: Status 404 returned error can't find the container with id a6ac5bc08e73d8cf42939ecb4f43b9bce3f0028d96536cd51f4e891bc7841b3a Jul 02 23:13:21.990111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.990034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77544c57b7-snjdl" event={"ID":"099ca819-e484-4e5b-a170-d108b401f9e8","Type":"ContainerStarted","Data":"c0a546cfeda75f94ed4ba77d115ef2864b0aec0b6090916589b0cc4a751f8ac8"} Jul 02 23:13:21.990111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:21.990088 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77544c57b7-snjdl" event={"ID":"099ca819-e484-4e5b-a170-d108b401f9e8","Type":"ContainerStarted","Data":"a6ac5bc08e73d8cf42939ecb4f43b9bce3f0028d96536cd51f4e891bc7841b3a"} Jul 02 23:13:22.010985 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:22.010927 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77544c57b7-snjdl" podStartSLOduration=1.010911185 podStartE2EDuration="1.010911185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:13:22.010195551 +0000 UTC m=+148.502387227" watchObservedRunningTime="2026-07-02 23:13:22.010911185 +0000 UTC m=+148.503102857" Jul 02 23:13:22.174591 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:22.174556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:22.231024 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:22.230985 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:26.780904 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.780870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:26.891192 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.891158 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:26.921894 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.921854 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:26.966033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.965999 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:13:26.968311 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.968294 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:26.977660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:26.977632 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:13:27.043041 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.042940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043041 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043038 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043099 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043117 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcv7\" (UniqueName: \"kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.043229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.043144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.084852 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.084808 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:27.134751 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.134724 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:27.143901 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.143875 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.143990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.143910 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.143990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.143929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.143990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.143945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.144148 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144004 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcv7\" (UniqueName: \"kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.144148 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.144148 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.144836 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144813 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.144947 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144882 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.145005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144980 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.145044 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.144987 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.146433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.146414 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.146646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.146629 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.152220 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.152192 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcv7\" (UniqueName: \"kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7\") pod \"console-57479d456d-5djvr\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.278612 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.278577 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:27.405707 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.405682 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:13:27.408255 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:13:27.408223 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7566e95_c2ed_4f58_a3fb_aad99e15c408.slice/crio-617aa78b89a903ffe399fb166ff6523aa6d397eef9fcfae04a5599e01842e2f0 WatchSource:0}: Error finding container 617aa78b89a903ffe399fb166ff6523aa6d397eef9fcfae04a5599e01842e2f0: Status 404 returned error can't find the container with id 617aa78b89a903ffe399fb166ff6523aa6d397eef9fcfae04a5599e01842e2f0 Jul 02 23:13:27.687996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:27.687907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:28.017609 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:28.017576 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57479d456d-5djvr" event={"ID":"b7566e95-c2ed-4f58-a3fb-aad99e15c408","Type":"ContainerStarted","Data":"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e"} Jul 02 23:13:28.018009 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:28.017635 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57479d456d-5djvr" event={"ID":"b7566e95-c2ed-4f58-a3fb-aad99e15c408","Type":"ContainerStarted","Data":"617aa78b89a903ffe399fb166ff6523aa6d397eef9fcfae04a5599e01842e2f0"} Jul 02 23:13:28.039508 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:28.039452 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57479d456d-5djvr" podStartSLOduration=2.039432907 podStartE2EDuration="2.039432907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:13:28.036746034 +0000 UTC m=+154.528937710" watchObservedRunningTime="2026-07-02 23:13:28.039432907 +0000 UTC m=+154.531624581" Jul 02 23:13:31.715044 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:31.715008 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:37.279514 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:37.279474 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:37.279921 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:37.279562 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:37.284201 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:37.284177 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:38.071229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.071202 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:13:38.118800 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.118762 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:13:38.221791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.221757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:38.277783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.277753 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:38.328893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.328825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:38.726140 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.726056 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:38.786558 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:38.786522 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:41.461580 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:41.461527 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:41.524982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:41.524945 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:51.052639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:51.052553 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:51.150315 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:51.150283 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:13:51.941885 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:51.941838 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-77544c57b7-snjdl" podUID="099ca819-e484-4e5b-a170-d108b401f9e8" containerName="console" containerID="cri-o://c0a546cfeda75f94ed4ba77d115ef2864b0aec0b6090916589b0cc4a751f8ac8" gracePeriod=15 Jul 02 23:13:52.118417 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.118381 2561 generic.go:356] "Generic (PLEG): container finished" podID="099ca819-e484-4e5b-a170-d108b401f9e8" containerID="c0a546cfeda75f94ed4ba77d115ef2864b0aec0b6090916589b0cc4a751f8ac8" exitCode=0 Jul 02 23:13:52.118768 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.118451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77544c57b7-snjdl" event={"ID":"099ca819-e484-4e5b-a170-d108b401f9e8","Type":"ContainerDied","Data":"c0a546cfeda75f94ed4ba77d115ef2864b0aec0b6090916589b0cc4a751f8ac8"} Jul 02 23:13:52.191850 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.191821 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:52.277783 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.277749 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qmfh\" (UniqueName: \"kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.277961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.277803 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.277961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.277840 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.277961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.277864 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.277961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.277895 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.278191 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278103 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.278191 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278154 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca\") pod \"099ca819-e484-4e5b-a170-d108b401f9e8\" (UID: \"099ca819-e484-4e5b-a170-d108b401f9e8\") " Jul 02 23:13:52.278401 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278371 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:13:52.278401 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278367 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:13:52.278559 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278506 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-oauth-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.278559 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278524 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-trusted-ca-bundle\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.278662 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278507 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config" (OuterVolumeSpecName: "console-config") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:13:52.278718 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.278676 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:13:52.279973 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.279955 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:13:52.280075 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.280002 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:13:52.280115 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.280076 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh" (OuterVolumeSpecName: "kube-api-access-5qmfh") pod "099ca819-e484-4e5b-a170-d108b401f9e8" (UID: "099ca819-e484-4e5b-a170-d108b401f9e8"). InnerVolumeSpecName "kube-api-access-5qmfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:13:52.379065 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.379022 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-console-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.379065 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.379055 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/099ca819-e484-4e5b-a170-d108b401f9e8-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.379065 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.379070 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmfh\" (UniqueName: \"kubernetes.io/projected/099ca819-e484-4e5b-a170-d108b401f9e8-kube-api-access-5qmfh\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.379298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.379081 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-oauth-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:52.379298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:52.379093 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/099ca819-e484-4e5b-a170-d108b401f9e8-console-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:13:53.124894 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:53.124863 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77544c57b7-snjdl" Jul 02 23:13:53.125387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:53.124862 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77544c57b7-snjdl" event={"ID":"099ca819-e484-4e5b-a170-d108b401f9e8","Type":"ContainerDied","Data":"a6ac5bc08e73d8cf42939ecb4f43b9bce3f0028d96536cd51f4e891bc7841b3a"} Jul 02 23:13:53.125387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:53.124992 2561 scope.go:117] "RemoveContainer" containerID="c0a546cfeda75f94ed4ba77d115ef2864b0aec0b6090916589b0cc4a751f8ac8" Jul 02 23:13:53.146758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:53.146725 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:53.152814 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:53.152783 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77544c57b7-snjdl"] Jul 02 23:13:54.068258 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:13:54.068224 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099ca819-e484-4e5b-a170-d108b401f9e8" path="/var/lib/kubelet/pods/099ca819-e484-4e5b-a170-d108b401f9e8/volumes" Jul 02 23:14:03.142863 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.142816 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6bbd88b6f6-k9d9n" podUID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" containerName="console" containerID="cri-o://3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499" gracePeriod=15 Jul 02 23:14:03.388028 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.388004 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:14:03.482804 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482713 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.482804 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482756 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.482804 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482796 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.483035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482840 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.483035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482864 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.483035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482888 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmblj\" (UniqueName: \"kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.483035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.482927 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca\") pod \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\" (UID: \"b3a8dc3b-4b24-4f64-9440-268ce87086f6\") " Jul 02 23:14:03.483219 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.483163 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:14:03.483375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.483342 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:14:03.483497 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.483388 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config" (OuterVolumeSpecName: "console-config") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:14:03.483497 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.483454 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:14:03.485152 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.485127 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:14:03.485251 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.485224 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:14:03.485251 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.485230 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj" (OuterVolumeSpecName: "kube-api-access-mmblj") pod "b3a8dc3b-4b24-4f64-9440-268ce87086f6" (UID: "b3a8dc3b-4b24-4f64-9440-268ce87086f6"). InnerVolumeSpecName "kube-api-access-mmblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:14:03.584643 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584606 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584643 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584637 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-oauth-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584643 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584647 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-trusted-ca-bundle\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584655 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584667 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-oauth-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584675 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a8dc3b-4b24-4f64-9440-268ce87086f6-console-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:03.584875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:03.584684 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmblj\" (UniqueName: \"kubernetes.io/projected/b3a8dc3b-4b24-4f64-9440-268ce87086f6-kube-api-access-mmblj\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:14:04.169492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.169459 2561 generic.go:356] "Generic (PLEG): container finished" podID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" containerID="3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499" exitCode=0 Jul 02 23:14:04.170042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.169516 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbd88b6f6-k9d9n" event={"ID":"b3a8dc3b-4b24-4f64-9440-268ce87086f6","Type":"ContainerDied","Data":"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499"} Jul 02 23:14:04.170042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.169585 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbd88b6f6-k9d9n" event={"ID":"b3a8dc3b-4b24-4f64-9440-268ce87086f6","Type":"ContainerDied","Data":"52caba4c5570a80e7422bada41ac61063a5e146657b29818582fb5ffc9da1f4b"} Jul 02 23:14:04.170042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.169519 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbd88b6f6-k9d9n" Jul 02 23:14:04.170042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.169611 2561 scope.go:117] "RemoveContainer" containerID="3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499" Jul 02 23:14:04.178064 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.178047 2561 scope.go:117] "RemoveContainer" containerID="3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499" Jul 02 23:14:04.178373 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:14:04.178351 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499\": container with ID starting with 3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499 not found: ID does not exist" containerID="3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499" Jul 02 23:14:04.178437 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.178383 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499"} err="failed to get container status \"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499\": rpc error: code = NotFound desc = could not find container \"3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499\": container with ID starting with 3c0bc69f38e8b18f14af9def39864239b16d9903e35aac5ac270800081218499 not found: ID does not exist" Jul 02 23:14:04.188981 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.188950 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:14:04.192767 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:04.192746 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bbd88b6f6-k9d9n"] Jul 02 23:14:06.068210 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:06.068177 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" path="/var/lib/kubelet/pods/b3a8dc3b-4b24-4f64-9440-268ce87086f6/volumes" Jul 02 23:14:06.642735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:06.642694 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:06.702544 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:06.702507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:27.129373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:27.129335 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:27.226453 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:27.226417 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:53.970210 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:53.970171 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:54.791496 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:54.791460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:57.568126 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:57.568086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:57.671872 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:57.671834 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:58.223161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:58.223122 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:59.073572 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:59.073520 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:59.216496 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:59.216464 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:59.665335 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:59.665297 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:14:59.755758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:14:59.755705 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:01.848365 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:01.848326 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:02.325939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:02.325908 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:04.392091 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:04.392058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:04.499214 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:04.499178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:06.815548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.815491 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct"] Jul 02 23:15:06.816009 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.815991 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" containerName="console" Jul 02 23:15:06.816054 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.816013 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" containerName="console" Jul 02 23:15:06.816087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.816064 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099ca819-e484-4e5b-a170-d108b401f9e8" containerName="console" Jul 02 23:15:06.816087 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.816073 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="099ca819-e484-4e5b-a170-d108b401f9e8" containerName="console" Jul 02 23:15:06.816152 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.816142 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a8dc3b-4b24-4f64-9440-268ce87086f6" containerName="console" Jul 02 23:15:06.816189 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.816158 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="099ca819-e484-4e5b-a170-d108b401f9e8" containerName="console" Jul 02 23:15:06.819478 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.819457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:06.827467 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.827447 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 23:15:06.828846 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.828827 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 23:15:06.829057 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.828826 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gwcx5\"" Jul 02 23:15:06.831047 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.831027 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct"] Jul 02 23:15:06.965274 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.965238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7t8\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-kube-api-access-gd7t8\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:06.965274 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:06.965284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.066341 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.066249 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7t8\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-kube-api-access-gd7t8\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.066341 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.066290 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.075588 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.075560 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7t8\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-kube-api-access-gd7t8\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.075588 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.075581 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec570b12-7dc8-41fc-8812-0761a6669152-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cf6ct\" (UID: \"ec570b12-7dc8-41fc-8812-0761a6669152\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.136724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.136687 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" Jul 02 23:15:07.268058 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.268005 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct"] Jul 02 23:15:07.270331 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:15:07.270306 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec570b12_7dc8_41fc_8812_0761a6669152.slice/crio-56068ed8318d1aa64074196512ab6f4070250daa9d73c8198d95b82dfe570f42 WatchSource:0}: Error finding container 56068ed8318d1aa64074196512ab6f4070250daa9d73c8198d95b82dfe570f42: Status 404 returned error can't find the container with id 56068ed8318d1aa64074196512ab6f4070250daa9d73c8198d95b82dfe570f42 Jul 02 23:15:07.424751 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:07.424662 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" event={"ID":"ec570b12-7dc8-41fc-8812-0761a6669152","Type":"ContainerStarted","Data":"56068ed8318d1aa64074196512ab6f4070250daa9d73c8198d95b82dfe570f42"} Jul 02 23:15:09.854372 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:09.854326 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:10.441984 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:10.441940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" event={"ID":"ec570b12-7dc8-41fc-8812-0761a6669152","Type":"ContainerStarted","Data":"f478e08590d7aad5cce7dc4cd47ca63219df03561da9868512b832b1ec8c0b83"} Jul 02 23:15:10.459603 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:10.459512 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cf6ct" podStartSLOduration=1.705792387 podStartE2EDuration="4.459489333s" totalImagesPullingTime="2.753696946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:15:10.457521555 +0000 UTC m=+256.949713234" watchObservedRunningTime="2026-07-02 23:15:10.459489333 +0000 UTC m=+256.951681009" Jul 02 23:15:10.775999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:10.775950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:10.883032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:10.882994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:11.923971 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:11.923935 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:12.023848 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:12.023819 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:13.759456 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.759420 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9"] Jul 02 23:15:13.761825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.761809 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:13.765108 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.765085 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 23:15:13.766685 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.766661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 23:15:13.766807 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.766723 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9d55p\"" Jul 02 23:15:13.776393 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.776362 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9"] Jul 02 23:15:13.934846 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.934810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47f7k\" (UniqueName: \"kubernetes.io/projected/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-kube-api-access-47f7k\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:13.934846 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:13.934852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.035453 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.035362 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47f7k\" (UniqueName: \"kubernetes.io/projected/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-kube-api-access-47f7k\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.035453 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.035401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.035878 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.035856 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.044527 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.044499 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47f7k\" (UniqueName: \"kubernetes.io/projected/780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc-kube-api-access-47f7k\") pod \"openshift-lws-operator-fd8ccff4c-dgwp9\" (UID: \"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.070540 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.070504 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" Jul 02 23:15:14.219100 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.219046 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9"] Jul 02 23:15:14.221938 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:15:14.221910 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod780fdb6c_0c2b_4f3d_8f30_bcb079cd02dc.slice/crio-cda08f71c060de67740d96a549151c732aec756b2cc5ee3fe928692d66ee9612 WatchSource:0}: Error finding container cda08f71c060de67740d96a549151c732aec756b2cc5ee3fe928692d66ee9612: Status 404 returned error can't find the container with id cda08f71c060de67740d96a549151c732aec756b2cc5ee3fe928692d66ee9612 Jul 02 23:15:14.460624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:14.460523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" event={"ID":"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc","Type":"ContainerStarted","Data":"cda08f71c060de67740d96a549151c732aec756b2cc5ee3fe928692d66ee9612"} Jul 02 23:15:18.483993 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:18.483954 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" event={"ID":"780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc","Type":"ContainerStarted","Data":"78474d46d19cfa473970822bee4bd07173ab05406acfad056c977275bf8a7c34"} Jul 02 23:15:18.516442 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:18.516395 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dgwp9" podStartSLOduration=2.21765312 podStartE2EDuration="5.516380395s" totalImagesPullingTime="3.298727275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:15:18.514089316 +0000 UTC m=+265.006280992" watchObservedRunningTime="2026-07-02 23:15:18.516380395 +0000 UTC m=+265.008572070" Jul 02 23:15:19.354764 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:19.354730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:19.455497 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:19.455465 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:21.254814 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:21.254782 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:33.563775 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:33.563740 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:34.918564 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.918517 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr"] Jul 02 23:15:34.922247 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.922223 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:34.938743 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.938712 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 23:15:34.941812 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.941791 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p65gt\"" Jul 02 23:15:34.942106 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.942088 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 23:15:34.953346 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.953320 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 23:15:34.953596 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.953375 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 23:15:34.955445 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:34.955413 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr"] Jul 02 23:15:35.014390 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.014354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.014585 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.014395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-tmp\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.014585 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.014479 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-manifests\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.014585 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.014514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.014704 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.014600 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfj5\" (UniqueName: \"kubernetes.io/projected/3f6a5972-1a24-457a-945d-98ae86d9a347-kube-api-access-ngfj5\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.115655 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.115613 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-manifests\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.115824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.115665 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.115824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.115709 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfj5\" (UniqueName: \"kubernetes.io/projected/3f6a5972-1a24-457a-945d-98ae86d9a347-kube-api-access-ngfj5\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.115824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.115730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.115824 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.115755 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-tmp\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.116111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.116087 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-manifests\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.118129 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.118096 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6a5972-1a24-457a-945d-98ae86d9a347-tmp\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.118264 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.118193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.118324 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.118262 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f6a5972-1a24-457a-945d-98ae86d9a347-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.129871 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.129849 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfj5\" (UniqueName: \"kubernetes.io/projected/3f6a5972-1a24-457a-945d-98ae86d9a347-kube-api-access-ngfj5\") pod \"opendatahub-operator-controller-manager-7fc9978976-nqdrr\" (UID: \"3f6a5972-1a24-457a-945d-98ae86d9a347\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.201699 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.201606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:35.232747 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.232709 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:35.405518 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:15:35.405488 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6a5972_1a24_457a_945d_98ae86d9a347.slice/crio-e098b9d0b0fbafbfe2ca67fadfb455c0b04a231fe106005847359d3299b25643 WatchSource:0}: Error finding container e098b9d0b0fbafbfe2ca67fadfb455c0b04a231fe106005847359d3299b25643: Status 404 returned error can't find the container with id e098b9d0b0fbafbfe2ca67fadfb455c0b04a231fe106005847359d3299b25643 Jul 02 23:15:35.413409 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.413386 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr"] Jul 02 23:15:35.431056 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.430999 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:35.555618 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.555590 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" event={"ID":"3f6a5972-1a24-457a-945d-98ae86d9a347","Type":"ContainerStarted","Data":"e098b9d0b0fbafbfe2ca67fadfb455c0b04a231fe106005847359d3299b25643"} Jul 02 23:15:35.663793 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.663760 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:35.797275 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.797220 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:35.978881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:35.978791 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:38.573529 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:38.573434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" event={"ID":"3f6a5972-1a24-457a-945d-98ae86d9a347","Type":"ContainerStarted","Data":"1ac02e53f121970d7561b3c6d5610b375d6e78da341209d2d706f99051a7494c"} Jul 02 23:15:39.580990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:39.580948 2561 generic.go:356] "Generic (PLEG): container finished" podID="3f6a5972-1a24-457a-945d-98ae86d9a347" containerID="1ac02e53f121970d7561b3c6d5610b375d6e78da341209d2d706f99051a7494c" exitCode=0 Jul 02 23:15:39.581519 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:39.581000 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" event={"ID":"3f6a5972-1a24-457a-945d-98ae86d9a347","Type":"ContainerDied","Data":"1ac02e53f121970d7561b3c6d5610b375d6e78da341209d2d706f99051a7494c"} Jul 02 23:15:39.751148 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:39.751115 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:39.986236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:39.986207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:40.588799 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:40.588762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" event={"ID":"3f6a5972-1a24-457a-945d-98ae86d9a347","Type":"ContainerStarted","Data":"fe1598eb6f698287bb3135f659325eb6000befa57d2f3596801ea7dcbe794e69"} Jul 02 23:15:40.589216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:40.588904 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:40.628453 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:40.628396 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" podStartSLOduration=2.2709869019999998 podStartE2EDuration="6.628377686s" totalImagesPullingTime="3.206910888s" totalInitContainerRuntime="1.150479896s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:15:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:15:40.627811542 +0000 UTC m=+287.120003218" watchObservedRunningTime="2026-07-02 23:15:40.628377686 +0000 UTC m=+287.120569360" Jul 02 23:15:51.595394 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:51.595359 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc9978976-nqdrr" Jul 02 23:15:53.911023 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:53.910995 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 23:15:54.007625 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:54.007593 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:54.265388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:54.265359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:55.240946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:55.240914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:55.507170 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:55.507125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:55.677983 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:55.677950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:56.697700 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:56.697663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:56.833744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:56.833715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:57.004902 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:57.004870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:57.435700 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:57.435611 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:58.771811 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:58.771777 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:58.862524 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:58.862491 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:15:59.960982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:15:59.960945 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:02.715288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.715253 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59"] Jul 02 23:16:02.719857 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.719835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.731068 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.731040 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 23:16:02.731197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.731089 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 23:16:02.731197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.731101 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 23:16:02.731197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.731098 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ct8dg\"" Jul 02 23:16:02.757426 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.757394 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59"] Jul 02 23:16:02.765273 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.765238 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:02.870620 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.870585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zbb\" (UniqueName: \"kubernetes.io/projected/bdc35939-6eb0-451f-b8e4-457c8eb85391-kube-api-access-69zbb\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.870813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.870634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.870813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.870680 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-metrics-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.870813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.870702 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc35939-6eb0-451f-b8e4-457c8eb85391-manager-config\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.950419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.950380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:02.971629 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.971528 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-metrics-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.971629 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.971608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc35939-6eb0-451f-b8e4-457c8eb85391-manager-config\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.971808 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.971691 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zbb\" (UniqueName: \"kubernetes.io/projected/bdc35939-6eb0-451f-b8e4-457c8eb85391-kube-api-access-69zbb\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.971808 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.971744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.972248 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.972224 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc35939-6eb0-451f-b8e4-457c8eb85391-manager-config\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.973957 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.973937 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-metrics-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.974132 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.974113 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc35939-6eb0-451f-b8e4-457c8eb85391-cert\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:02.988908 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:02.988879 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zbb\" (UniqueName: \"kubernetes.io/projected/bdc35939-6eb0-451f-b8e4-457c8eb85391-kube-api-access-69zbb\") pod \"lws-controller-manager-9d6d7b94f-xqj59\" (UID: \"bdc35939-6eb0-451f-b8e4-457c8eb85391\") " pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:03.028791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.028752 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:03.211482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.211445 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh"] Jul 02 23:16:03.215045 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.215024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.219384 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219359 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-tls\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" Jul 02 23:16:03.219491 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219383 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istiod-openshift-gateway-dockercfg-rrf95\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rrf95\"" type="*v1.Secret" Jul 02 23:16:03.219491 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219418 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" podUID="49bd5357-62f1-44b9-a4f8-cf4c51e2eb27" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.219491 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219432 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 02 23:16:03.219639 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219585 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" Jul 02 23:16:03.219911 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.219890 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cacerts\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" Jul 02 23:16:03.220061 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.220033 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"istio-kubeconfig\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" Jul 02 23:16:03.220246 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:03.220225 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-16.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-16.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 02 23:16:03.237322 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.237258 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59"] Jul 02 23:16:03.254606 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:16:03.254569 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc35939_6eb0_451f_b8e4_457c8eb85391.slice/crio-2646e9c8378944d44b47ca433ccb78d46275b0389964332458df14ed45815e2e WatchSource:0}: Error finding container 2646e9c8378944d44b47ca433ccb78d46275b0389964332458df14ed45815e2e: Status 404 returned error can't find the container with id 2646e9c8378944d44b47ca433ccb78d46275b0389964332458df14ed45815e2e Jul 02 23:16:03.256768 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.256750 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:16:03.295566 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.295516 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh"] Jul 02 23:16:03.374884 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.374841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375055 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.374928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375055 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.374959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375055 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.375026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.375059 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzmf\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-kube-api-access-hlzmf\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.375085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.375223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.375106 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.476766 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.476664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.476939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.476846 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.476939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.476893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.477048 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.476968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.477048 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.477004 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzmf\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-kube-api-access-hlzmf\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.477048 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.477029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.477187 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.477056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.480133 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.480098 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.505111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.505076 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:03.700653 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:03.700613 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" event={"ID":"bdc35939-6eb0-451f-b8e4-457c8eb85391","Type":"ContainerStarted","Data":"2646e9c8378944d44b47ca433ccb78d46275b0389964332458df14ed45815e2e"} Jul 02 23:16:04.242706 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.242667 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 23:16:04.398772 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.398732 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 23:16:04.410958 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.410925 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:04.451522 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.451487 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 23:16:04.461652 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.461605 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:04.477304 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:04.477271 2561 configmap.go:193] Couldn't get configMap openshift-ingress/openshift-gw-ca-root-cert: failed to sync configmap cache: timed out waiting for the condition Jul 02 23:16:04.477474 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:04.477364 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap podName:49bd5357-62f1-44b9-a4f8-cf4c51e2eb27 nodeName:}" failed. No retries permitted until 2026-07-02 23:16:04.977342571 +0000 UTC m=+311.469534239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "istio-csr-ca-configmap" (UniqueName: "kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap") pod "istiod-openshift-gateway-94bb8fbfd-vh6bh" (UID: "49bd5357-62f1-44b9-a4f8-cf4c51e2eb27") : failed to sync configmap cache: timed out waiting for the condition Jul 02 23:16:04.477643 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:04.477626 2561 secret.go:189] Couldn't get secret openshift-ingress/cacerts: failed to sync secret cache: timed out waiting for the condition Jul 02 23:16:04.477720 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:16:04.477704 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts podName:49bd5357-62f1-44b9-a4f8-cf4c51e2eb27 nodeName:}" failed. No retries permitted until 2026-07-02 23:16:04.977683687 +0000 UTC m=+311.469875354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cacerts" (UniqueName: "kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts") pod "istiod-openshift-gateway-94bb8fbfd-vh6bh" (UID: "49bd5357-62f1-44b9-a4f8-cf4c51e2eb27") : failed to sync secret cache: timed out waiting for the condition Jul 02 23:16:04.595646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.595611 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 23:16:04.615318 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.615288 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 23:16:04.696737 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.696689 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:04.754668 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.754635 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 23:16:04.759944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.759920 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzmf\" (UniqueName: \"kubernetes.io/projected/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-kube-api-access-hlzmf\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:04.810041 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.810014 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rrf95\"" Jul 02 23:16:04.818819 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.818785 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:04.916470 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:04.916367 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:05.002149 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.002118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:05.002332 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.002161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:05.002841 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.002810 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:05.004349 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.004328 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49bd5357-62f1-44b9-a4f8-cf4c51e2eb27-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vh6bh\" (UID: \"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:05.027690 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.027643 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:05.189958 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.189836 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh"] Jul 02 23:16:05.192621 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:16:05.192586 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bd5357_62f1_44b9_a4f8_cf4c51e2eb27.slice/crio-7fa8770c1e5dcf6b7dc5fcb22a6e010f0d62e3a7ac3721b41cf6e6a17280bb38 WatchSource:0}: Error finding container 7fa8770c1e5dcf6b7dc5fcb22a6e010f0d62e3a7ac3721b41cf6e6a17280bb38: Status 404 returned error can't find the container with id 7fa8770c1e5dcf6b7dc5fcb22a6e010f0d62e3a7ac3721b41cf6e6a17280bb38 Jul 02 23:16:05.710829 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:05.710767 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" event={"ID":"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27","Type":"ContainerStarted","Data":"7fa8770c1e5dcf6b7dc5fcb22a6e010f0d62e3a7ac3721b41cf6e6a17280bb38"} Jul 02 23:16:06.196125 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:06.196083 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:06.374146 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:06.374088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:06.704193 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:06.704163 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:07.185758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:07.185723 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:07.721863 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:07.721826 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" event={"ID":"bdc35939-6eb0-451f-b8e4-457c8eb85391","Type":"ContainerStarted","Data":"2a2ff49636f74ff5c9e13321afc2f95ce4c984e160b78b70513927b2293c69a0"} Jul 02 23:16:07.722028 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:07.721922 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:07.774473 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:07.774413 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" podStartSLOduration=2.363051247 podStartE2EDuration="5.774395068s" totalImagesPullingTime="3.411343821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:16:07.773486965 +0000 UTC m=+314.265678640" watchObservedRunningTime="2026-07-02 23:16:07.774395068 +0000 UTC m=+314.266586743" Jul 02 23:16:07.814679 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:07.814642 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:08.061828 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:08.061793 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:08.226860 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:08.226827 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:10.719959 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:10.719918 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 02 23:16:10.720269 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:10.719991 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 02 23:16:11.745250 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:11.745207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" event={"ID":"49bd5357-62f1-44b9-a4f8-cf4c51e2eb27","Type":"ContainerStarted","Data":"528ae14d17e4b1cc1361af224f7c12a1322a860ff7f13023237f5566879f58f3"} Jul 02 23:16:11.745765 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:11.745439 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:11.747153 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:11.747118 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vh6bh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 23:16:11.747268 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:11.747182 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" podUID="49bd5357-62f1-44b9-a4f8-cf4c51e2eb27" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 23:16:11.801511 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:11.801452 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" podStartSLOduration=3.2766132519999998 podStartE2EDuration="8.801435731s" totalImagesPullingTime="5.524822479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:16:11.798890695 +0000 UTC m=+318.291082370" watchObservedRunningTime="2026-07-02 23:16:11.801435731 +0000 UTC m=+318.293627405" Jul 02 23:16:12.750077 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:12.750049 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vh6bh" Jul 02 23:16:18.731674 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:18.731641 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9d6d7b94f-xqj59" Jul 02 23:16:30.024298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:30.024263 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:30.119740 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:30.119664 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:53.011753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:53.011668 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:55.431351 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:55.431315 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:55.531269 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:55.531222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:55.672712 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:55.672685 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:56.364362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:56.364328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:16:56.555364 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:16:56.555333 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:21.385664 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:21.385632 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:24.853184 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:24.853150 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:24.991489 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:24.991449 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:29.642178 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:29.642131 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:30.010706 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:30.010666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:31.169571 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:31.169522 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:33.259188 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:33.259111 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:33.876428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:33.876396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:34.059289 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:34.059254 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:35.201033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:35.201002 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:36.945107 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:36.945070 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:37.312955 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:37.312895 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:37.430399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:37.430373 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:37.521759 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:37.521721 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:40.570883 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:40.570844 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:40.646126 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:40.646084 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:49.671139 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:49.671102 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:49.773469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:49.773421 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:50.225230 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:50.225172 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:50.430697 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:50.430659 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:54.207034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:54.207001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:54.298192 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:54.298155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:59.674899 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:59.674864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:17:59.847415 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:17:59.847386 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:03.186601 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:03.186558 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:03.318541 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:03.318493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:12.762665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:12.762628 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:12.863095 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:12.863060 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:15.217212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:15.217166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:15.317385 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:15.317349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:22.384142 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:22.384088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:22.486202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:22.486170 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:23.317501 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.317471 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb9cc4588-n2xfj"] Jul 02 23:18:23.320798 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.320778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.340743 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.340716 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb9cc4588-n2xfj"] Jul 02 23:18:23.394522 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.394483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:23.404654 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrpj\" (UniqueName: \"kubernetes.io/projected/3417fe17-cdbc-4d05-9239-97ff4acb3d67-kube-api-access-gmrpj\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404737 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-trusted-ca-bundle\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404764 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-service-ca\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404795 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-oauth-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404916 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.404916 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.404870 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-oauth-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.458762 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.458728 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:23.506033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.505988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-service-ca\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-oauth-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-oauth-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrpj\" (UniqueName: \"kubernetes.io/projected/3417fe17-cdbc-4d05-9239-97ff4acb3d67-kube-api-access-gmrpj\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506237 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506263 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-trusted-ca-bundle\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506897 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506875 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-oauth-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506988 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506877 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-service-ca\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.506988 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.506916 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.507259 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.507237 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3417fe17-cdbc-4d05-9239-97ff4acb3d67-trusted-ca-bundle\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.509085 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.509062 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-oauth-config\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.509228 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.509212 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3417fe17-cdbc-4d05-9239-97ff4acb3d67-console-serving-cert\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.516322 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.516289 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrpj\" (UniqueName: \"kubernetes.io/projected/3417fe17-cdbc-4d05-9239-97ff4acb3d67-kube-api-access-gmrpj\") pod \"console-bb9cc4588-n2xfj\" (UID: \"3417fe17-cdbc-4d05-9239-97ff4acb3d67\") " pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.630502 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.630406 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:23.771570 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:23.771525 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb9cc4588-n2xfj"] Jul 02 23:18:23.773296 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:18:23.773267 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3417fe17_cdbc_4d05_9239_97ff4acb3d67.slice/crio-b50ee05593d1d4d08c578bb332142053d5c2348b449c93b67fd0bbc691336761 WatchSource:0}: Error finding container b50ee05593d1d4d08c578bb332142053d5c2348b449c93b67fd0bbc691336761: Status 404 returned error can't find the container with id b50ee05593d1d4d08c578bb332142053d5c2348b449c93b67fd0bbc691336761 Jul 02 23:18:24.372050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:24.372016 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb9cc4588-n2xfj" event={"ID":"3417fe17-cdbc-4d05-9239-97ff4acb3d67","Type":"ContainerStarted","Data":"75aaea613c0afba50d13e63a41f07cc52e025e3eb285a72e5f276bbdf1775e10"} Jul 02 23:18:24.372275 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:24.372068 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb9cc4588-n2xfj" event={"ID":"3417fe17-cdbc-4d05-9239-97ff4acb3d67","Type":"ContainerStarted","Data":"b50ee05593d1d4d08c578bb332142053d5c2348b449c93b67fd0bbc691336761"} Jul 02 23:18:24.393964 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:24.393905 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb9cc4588-n2xfj" podStartSLOduration=1.393887136 podStartE2EDuration="1.393887136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:18:24.392615555 +0000 UTC m=+450.884807229" watchObservedRunningTime="2026-07-02 23:18:24.393887136 +0000 UTC m=+450.886078811" Jul 02 23:18:27.283222 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:27.283185 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:27.523524 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:27.523475 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:29.007205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.007170 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:29.018458 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.018425 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:29.021944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.021926 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.024667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.024641 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 23:18:29.024667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.024656 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rrj7j\"" Jul 02 23:18:29.024938 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.024918 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 23:18:29.026091 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.026076 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 23:18:29.029985 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.029827 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:29.111412 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.111368 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:29.129218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.129189 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:29.162746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.162716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.162933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.162780 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb2z\" (UniqueName: \"kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.230352 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.230323 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:29.263933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.263854 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.263933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.263918 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb2z\" (UniqueName: \"kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.264487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.264466 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.273292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.273257 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb2z\" (UniqueName: \"kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z\") pod \"limitador-limitador-64c94fd64f-v9vmq\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.334421 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.334386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:29.473933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.473903 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:29.476941 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:18:29.476910 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0a203c_c062_467d_9abd_bfb50b058f02.slice/crio-64138e1ce33c0c02c0f2c6c8f5b024e2a1631d9d546c2f36faa617d1958f7203 WatchSource:0}: Error finding container 64138e1ce33c0c02c0f2c6c8f5b024e2a1631d9d546c2f36faa617d1958f7203: Status 404 returned error can't find the container with id 64138e1ce33c0c02c0f2c6c8f5b024e2a1631d9d546c2f36faa617d1958f7203 Jul 02 23:18:29.846482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.846446 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:29.932922 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:29.932887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:30.403096 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:30.403055 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" event={"ID":"8b0a203c-c062-467d-9abd-bfb50b058f02","Type":"ContainerStarted","Data":"64138e1ce33c0c02c0f2c6c8f5b024e2a1631d9d546c2f36faa617d1958f7203"} Jul 02 23:18:32.416071 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:32.415979 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" event={"ID":"8b0a203c-c062-467d-9abd-bfb50b058f02","Type":"ContainerStarted","Data":"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551"} Jul 02 23:18:32.416452 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:32.416116 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:32.435659 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:32.435604 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" podStartSLOduration=0.856906579 podStartE2EDuration="3.435587982s" totalImagesPullingTime="2.578681403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:18:32.434704663 +0000 UTC m=+458.926896338" watchObservedRunningTime="2026-07-02 23:18:32.435587982 +0000 UTC m=+458.927779680" Jul 02 23:18:33.631267 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:33.631229 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:33.631697 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:33.631303 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:33.636033 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:33.636012 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:34.428901 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:34.428873 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb9cc4588-n2xfj" Jul 02 23:18:34.478300 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:34.478265 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:18:34.578157 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:34.578125 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:34.635902 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:34.635866 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:43.422667 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:43.422635 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:45.812962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:45.812923 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:45.813481 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:45.813125 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" podUID="8b0a203c-c062-467d-9abd-bfb50b058f02" containerName="limitador" containerID="cri-o://1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551" gracePeriod=30 Jul 02 23:18:46.355875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.355847 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:46.489368 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.489285 2561 generic.go:356] "Generic (PLEG): container finished" podID="8b0a203c-c062-467d-9abd-bfb50b058f02" containerID="1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551" exitCode=0 Jul 02 23:18:46.489368 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.489329 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" Jul 02 23:18:46.489573 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.489370 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" event={"ID":"8b0a203c-c062-467d-9abd-bfb50b058f02","Type":"ContainerDied","Data":"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551"} Jul 02 23:18:46.489573 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.489425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-v9vmq" event={"ID":"8b0a203c-c062-467d-9abd-bfb50b058f02","Type":"ContainerDied","Data":"64138e1ce33c0c02c0f2c6c8f5b024e2a1631d9d546c2f36faa617d1958f7203"} Jul 02 23:18:46.489573 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.489452 2561 scope.go:117] "RemoveContainer" containerID="1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551" Jul 02 23:18:46.497847 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.497827 2561 scope.go:117] "RemoveContainer" containerID="1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551" Jul 02 23:18:46.498109 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:18:46.498090 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551\": container with ID starting with 1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551 not found: ID does not exist" containerID="1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551" Jul 02 23:18:46.498159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.498119 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551"} err="failed to get container status \"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551\": rpc error: code = NotFound desc = could not find container \"1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551\": container with ID starting with 1f59ad51da3a9312fc6e63cdfb5822cd2b4e2eefb25cc5a390a3a1f66006f551 not found: ID does not exist" Jul 02 23:18:46.535482 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.535447 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlb2z\" (UniqueName: \"kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z\") pod \"8b0a203c-c062-467d-9abd-bfb50b058f02\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " Jul 02 23:18:46.535658 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.535564 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file\") pod \"8b0a203c-c062-467d-9abd-bfb50b058f02\" (UID: \"8b0a203c-c062-467d-9abd-bfb50b058f02\") " Jul 02 23:18:46.535931 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.535907 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file" (OuterVolumeSpecName: "config-file") pod "8b0a203c-c062-467d-9abd-bfb50b058f02" (UID: "8b0a203c-c062-467d-9abd-bfb50b058f02"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:18:46.537748 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.537723 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z" (OuterVolumeSpecName: "kube-api-access-vlb2z") pod "8b0a203c-c062-467d-9abd-bfb50b058f02" (UID: "8b0a203c-c062-467d-9abd-bfb50b058f02"). InnerVolumeSpecName "kube-api-access-vlb2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:18:46.637215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.637175 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlb2z\" (UniqueName: \"kubernetes.io/projected/8b0a203c-c062-467d-9abd-bfb50b058f02-kube-api-access-vlb2z\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:46.637215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.637210 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b0a203c-c062-467d-9abd-bfb50b058f02-config-file\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:46.811873 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.811840 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:46.817103 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:46.817070 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-v9vmq"] Jul 02 23:18:48.068969 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:48.068931 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b0a203c-c062-467d-9abd-bfb50b058f02" path="/var/lib/kubelet/pods/8b0a203c-c062-467d-9abd-bfb50b058f02/volumes" Jul 02 23:18:50.548375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.548342 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-nncqg"] Jul 02 23:18:50.549066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.548899 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0a203c-c062-467d-9abd-bfb50b058f02" containerName="limitador" Jul 02 23:18:50.549066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.548918 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0a203c-c062-467d-9abd-bfb50b058f02" containerName="limitador" Jul 02 23:18:50.549066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.548990 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0a203c-c062-467d-9abd-bfb50b058f02" containerName="limitador" Jul 02 23:18:50.553399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.553369 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.556347 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.556314 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 23:18:50.556481 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.556328 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qq2pp\"" Jul 02 23:18:50.562275 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.562248 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-nncqg"] Jul 02 23:18:50.572164 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.572109 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c92\" (UniqueName: \"kubernetes.io/projected/3406c55b-22af-46b8-a49d-d505611cf820-kube-api-access-z9c92\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.572367 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.572315 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3406c55b-22af-46b8-a49d-d505611cf820-data\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.587996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.587960 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:18:50.673057 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.673020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3406c55b-22af-46b8-a49d-d505611cf820-data\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.673274 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.673085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c92\" (UniqueName: \"kubernetes.io/projected/3406c55b-22af-46b8-a49d-d505611cf820-kube-api-access-z9c92\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.673498 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.673476 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3406c55b-22af-46b8-a49d-d505611cf820-data\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.682126 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.682090 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c92\" (UniqueName: \"kubernetes.io/projected/3406c55b-22af-46b8-a49d-d505611cf820-kube-api-access-z9c92\") pod \"postgres-69fb4594cf-nncqg\" (UID: \"3406c55b-22af-46b8-a49d-d505611cf820\") " pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.866402 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.866310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:50.999176 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:50.999147 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-nncqg"] Jul 02 23:18:51.001275 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:18:51.001240 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3406c55b_22af_46b8_a49d_d505611cf820.slice/crio-0a51e72ab374192616c8dc42214533fef598e2f84c70e63c42819908222a4ce0 WatchSource:0}: Error finding container 0a51e72ab374192616c8dc42214533fef598e2f84c70e63c42819908222a4ce0: Status 404 returned error can't find the container with id 0a51e72ab374192616c8dc42214533fef598e2f84c70e63c42819908222a4ce0 Jul 02 23:18:51.516985 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:51.516950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-nncqg" event={"ID":"3406c55b-22af-46b8-a49d-d505611cf820","Type":"ContainerStarted","Data":"0a51e72ab374192616c8dc42214533fef598e2f84c70e63c42819908222a4ce0"} Jul 02 23:18:56.237554 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:56.237506 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 23:18:56.545738 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:56.545695 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-nncqg" event={"ID":"3406c55b-22af-46b8-a49d-d505611cf820","Type":"ContainerStarted","Data":"282a0b36fc3eb21a8ffce0136a5e35c7f040399b42c2ab467da5bfe10b674fc8"} Jul 02 23:18:56.545939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:56.545811 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:18:56.563977 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:56.563919 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-nncqg" podStartSLOduration=1.332586958 podStartE2EDuration="6.563904523s" totalImagesPullingTime="5.231317565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:18:56.561629049 +0000 UTC m=+483.053820724" watchObservedRunningTime="2026-07-02 23:18:56.563904523 +0000 UTC m=+483.056096198" Jul 02 23:18:59.499338 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.499290 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-57479d456d-5djvr" podUID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" containerName="console" containerID="cri-o://b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e" gracePeriod=15 Jul 02 23:18:59.747030 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.747006 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:18:59.861076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.860973 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861039 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861076 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861132 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwcv7\" (UniqueName: \"kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861173 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861253 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861285 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle\") pod \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\" (UID: \"b7566e95-c2ed-4f58-a3fb-aad99e15c408\") " Jul 02 23:18:59.861696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861656 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config" (OuterVolumeSpecName: "console-config") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:18:59.861797 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861699 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:18:59.861920 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861890 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:18:59.862000 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.861979 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:18:59.863359 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.863332 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:18:59.863452 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.863377 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:18:59.863452 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.863390 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7" (OuterVolumeSpecName: "kube-api-access-wwcv7") pod "b7566e95-c2ed-4f58-a3fb-aad99e15c408" (UID: "b7566e95-c2ed-4f58-a3fb-aad99e15c408"). InnerVolumeSpecName "kube-api-access-wwcv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:18:59.962881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962842 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-oauth-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.962881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962874 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcv7\" (UniqueName: \"kubernetes.io/projected/b7566e95-c2ed-4f58-a3fb-aad99e15c408-kube-api-access-wwcv7\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.962881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962883 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-oauth-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.963161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962892 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-config\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.963161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962901 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-trusted-ca-bundle\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.963161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962909 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7566e95-c2ed-4f58-a3fb-aad99e15c408-console-serving-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:18:59.963161 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:18:59.962917 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7566e95-c2ed-4f58-a3fb-aad99e15c408-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:19:00.567383 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.567346 2561 generic.go:356] "Generic (PLEG): container finished" podID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" containerID="b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e" exitCode=0 Jul 02 23:19:00.567893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.567394 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57479d456d-5djvr" Jul 02 23:19:00.567893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.567409 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57479d456d-5djvr" event={"ID":"b7566e95-c2ed-4f58-a3fb-aad99e15c408","Type":"ContainerDied","Data":"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e"} Jul 02 23:19:00.567893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.567451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57479d456d-5djvr" event={"ID":"b7566e95-c2ed-4f58-a3fb-aad99e15c408","Type":"ContainerDied","Data":"617aa78b89a903ffe399fb166ff6523aa6d397eef9fcfae04a5599e01842e2f0"} Jul 02 23:19:00.567893 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.567477 2561 scope.go:117] "RemoveContainer" containerID="b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e" Jul 02 23:19:00.576269 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.576251 2561 scope.go:117] "RemoveContainer" containerID="b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e" Jul 02 23:19:00.576602 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:19:00.576578 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e\": container with ID starting with b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e not found: ID does not exist" containerID="b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e" Jul 02 23:19:00.576679 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.576615 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e"} err="failed to get container status \"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e\": rpc error: code = NotFound desc = could not find container \"b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e\": container with ID starting with b560f76fbb6a4a1e709ddd4eb6026923d1582ed1ec43755291cc2ff1dc3f342e not found: ID does not exist" Jul 02 23:19:00.586519 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.586472 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:19:00.591351 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:00.591316 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57479d456d-5djvr"] Jul 02 23:19:02.068756 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:02.068715 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" path="/var/lib/kubelet/pods/b7566e95-c2ed-4f58-a3fb-aad99e15c408/volumes" Jul 02 23:19:02.581717 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:02.581685 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-nncqg" Jul 02 23:19:02.888721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:02.888636 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:03.250112 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:03.250084 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:09.569491 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.569453 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:09.569915 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.569855 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" containerName="console" Jul 02 23:19:09.569915 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.569870 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" containerName="console" Jul 02 23:19:09.569989 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.569962 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7566e95-c2ed-4f58-a3fb-aad99e15c408" containerName="console" Jul 02 23:19:09.581251 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.581217 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:09.591306 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.591275 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.591496 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.591409 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:09.594761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.594682 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 23:19:09.594761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.594732 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d4hs2\"" Jul 02 23:19:09.594761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.594746 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 23:19:09.655818 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.655787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrjc\" (UniqueName: \"kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.655999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.655866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.673774 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.673744 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:09.744067 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.744030 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:09.750679 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.750652 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:19:09.753240 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.753221 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.756796 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.756777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrjc\" (UniqueName: \"kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.756915 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.756833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.756981 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:19:09.756946 2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 02 23:19:09.757036 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:19:09.757001 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert podName:60e57e67-95cc-4138-85cf-d80899a083ab nodeName:}" failed. No retries permitted until 2026-07-02 23:19:10.256983424 +0000 UTC m=+496.749175078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert") pod "maas-controller-8648b6f8d4-dgpxd" (UID: "60e57e67-95cc-4138-85cf-d80899a083ab") : secret "maas-controller-webhook-cert" not found Jul 02 23:19:09.763419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.763393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:19:09.772255 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.772230 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrjc\" (UniqueName: \"kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:09.857882 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.857785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74z6\" (UniqueName: \"kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.857882 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.857833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.889982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.889952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:09.959325 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.959282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74z6\" (UniqueName: \"kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.959505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.959335 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.961972 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.961946 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:09.967679 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:09.967658 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74z6\" (UniqueName: \"kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6\") pod \"maas-controller-fccbb966b-pchs7\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:10.064885 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.064848 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:10.203617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.203578 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:19:10.206424 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:19:10.206394 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20934018_c9d3_4698_9df3_6fea7a2fe626.slice/crio-f9daf680019044bf98585e6ca5f2ff6b7bab10107fe2d9a56417336fee60dd38 WatchSource:0}: Error finding container f9daf680019044bf98585e6ca5f2ff6b7bab10107fe2d9a56417336fee60dd38: Status 404 returned error can't find the container with id f9daf680019044bf98585e6ca5f2ff6b7bab10107fe2d9a56417336fee60dd38 Jul 02 23:19:10.261879 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.261831 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:10.264348 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.264330 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") pod \"maas-controller-8648b6f8d4-dgpxd\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:10.504367 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.504325 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:10.619127 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.619093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-pchs7" event={"ID":"20934018-c9d3-4698-9df3-6fea7a2fe626","Type":"ContainerStarted","Data":"f9daf680019044bf98585e6ca5f2ff6b7bab10107fe2d9a56417336fee60dd38"} Jul 02 23:19:10.637746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:10.637722 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:10.639447 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:19:10.639424 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e57e67_95cc_4138_85cf_d80899a083ab.slice/crio-b469eef190dd775edfd88477d25c4e43ccac419f3711400b1bbdf36bc77c074e WatchSource:0}: Error finding container b469eef190dd775edfd88477d25c4e43ccac419f3711400b1bbdf36bc77c074e: Status 404 returned error can't find the container with id b469eef190dd775edfd88477d25c4e43ccac419f3711400b1bbdf36bc77c074e Jul 02 23:19:11.627376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:11.627334 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" event={"ID":"60e57e67-95cc-4138-85cf-d80899a083ab","Type":"ContainerStarted","Data":"b469eef190dd775edfd88477d25c4e43ccac419f3711400b1bbdf36bc77c074e"} Jul 02 23:19:12.400374 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.400340 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:12.483709 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.483634 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:12.633916 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.633877 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" event={"ID":"60e57e67-95cc-4138-85cf-d80899a083ab","Type":"ContainerStarted","Data":"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07"} Jul 02 23:19:12.634402 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.633939 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:12.635549 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.635511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-pchs7" event={"ID":"20934018-c9d3-4698-9df3-6fea7a2fe626","Type":"ContainerStarted","Data":"02dfa20f36d2bc62fa606f32c090a672107a4117470530fac3e2335a88964548"} Jul 02 23:19:12.635677 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.635593 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:12.654914 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.654865 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" podStartSLOduration=2.099602613 podStartE2EDuration="3.65485183s" totalImagesPullingTime="1.555249217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:19:12.652222555 +0000 UTC m=+499.144414240" watchObservedRunningTime="2026-07-02 23:19:12.65485183 +0000 UTC m=+499.147043531" Jul 02 23:19:12.672126 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:12.672079 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fccbb966b-pchs7" podStartSLOduration=1.688477129 podStartE2EDuration="3.672065457s" totalImagesPullingTime="1.983588328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:19:12.670315009 +0000 UTC m=+499.162506684" watchObservedRunningTime="2026-07-02 23:19:12.672065457 +0000 UTC m=+499.164257132" Jul 02 23:19:23.645203 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:23.645172 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:19:23.645624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:23.645343 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:23.717357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:23.717325 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:23.717521 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:23.717502 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" podUID="60e57e67-95cc-4138-85cf-d80899a083ab" containerName="manager" containerID="cri-o://93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07" gracePeriod=30 Jul 02 23:19:23.962408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:23.962381 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:24.097285 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.097240 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbrjc\" (UniqueName: \"kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc\") pod \"60e57e67-95cc-4138-85cf-d80899a083ab\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " Jul 02 23:19:24.097460 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.097337 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") pod \"60e57e67-95cc-4138-85cf-d80899a083ab\" (UID: \"60e57e67-95cc-4138-85cf-d80899a083ab\") " Jul 02 23:19:24.099598 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.099562 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "60e57e67-95cc-4138-85cf-d80899a083ab" (UID: "60e57e67-95cc-4138-85cf-d80899a083ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:19:24.099598 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.099568 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc" (OuterVolumeSpecName: "kube-api-access-xbrjc") pod "60e57e67-95cc-4138-85cf-d80899a083ab" (UID: "60e57e67-95cc-4138-85cf-d80899a083ab"). InnerVolumeSpecName "kube-api-access-xbrjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:19:24.198887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.198783 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60e57e67-95cc-4138-85cf-d80899a083ab-webhook-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:19:24.198887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.198828 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrjc\" (UniqueName: \"kubernetes.io/projected/60e57e67-95cc-4138-85cf-d80899a083ab-kube-api-access-xbrjc\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:19:24.696445 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.696406 2561 generic.go:356] "Generic (PLEG): container finished" podID="60e57e67-95cc-4138-85cf-d80899a083ab" containerID="93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07" exitCode=0 Jul 02 23:19:24.696891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.696467 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" Jul 02 23:19:24.696891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.696485 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" event={"ID":"60e57e67-95cc-4138-85cf-d80899a083ab","Type":"ContainerDied","Data":"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07"} Jul 02 23:19:24.696891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.696559 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8648b6f8d4-dgpxd" event={"ID":"60e57e67-95cc-4138-85cf-d80899a083ab","Type":"ContainerDied","Data":"b469eef190dd775edfd88477d25c4e43ccac419f3711400b1bbdf36bc77c074e"} Jul 02 23:19:24.696891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.696591 2561 scope.go:117] "RemoveContainer" containerID="93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07" Jul 02 23:19:24.706227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.706206 2561 scope.go:117] "RemoveContainer" containerID="93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07" Jul 02 23:19:24.706520 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:19:24.706501 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07\": container with ID starting with 93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07 not found: ID does not exist" containerID="93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07" Jul 02 23:19:24.706595 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.706550 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07"} err="failed to get container status \"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07\": rpc error: code = NotFound desc = could not find container \"93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07\": container with ID starting with 93c33b5aff1a6a66581953b8cca764caacc00f634fb043edb61a4340de18fb07 not found: ID does not exist" Jul 02 23:19:24.719302 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.719268 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:24.725922 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:24.725894 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8648b6f8d4-dgpxd"] Jul 02 23:19:26.068297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:26.068263 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60e57e67-95cc-4138-85cf-d80899a083ab" path="/var/lib/kubelet/pods/60e57e67-95cc-4138-85cf-d80899a083ab/volumes" Jul 02 23:19:43.969744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:43.969705 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:44.572024 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.571977 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:44.610774 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.610740 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5595cc7ddb-qxc92"] Jul 02 23:19:44.611134 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.611121 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60e57e67-95cc-4138-85cf-d80899a083ab" containerName="manager" Jul 02 23:19:44.611188 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.611135 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60e57e67-95cc-4138-85cf-d80899a083ab" containerName="manager" Jul 02 23:19:44.611225 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.611190 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60e57e67-95cc-4138-85cf-d80899a083ab" containerName="manager" Jul 02 23:19:44.614727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.614705 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.617927 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.617906 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 02 23:19:44.618056 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.617968 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tnxm4\"" Jul 02 23:19:44.625682 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.625659 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5595cc7ddb-qxc92"] Jul 02 23:19:44.685360 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.685327 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:44.697923 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.697890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a52401-27c3-4577-916b-e7c7d902536a-maas-api-tls\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.698086 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.697952 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mm5\" (UniqueName: \"kubernetes.io/projected/01a52401-27c3-4577-916b-e7c7d902536a-kube-api-access-z6mm5\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.698086 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.698011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a52401-27c3-4577-916b-e7c7d902536a-openshift-service-ca\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.799209 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.799181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a52401-27c3-4577-916b-e7c7d902536a-maas-api-tls\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.799388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.799228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mm5\" (UniqueName: \"kubernetes.io/projected/01a52401-27c3-4577-916b-e7c7d902536a-kube-api-access-z6mm5\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.799388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.799251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a52401-27c3-4577-916b-e7c7d902536a-openshift-service-ca\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.799941 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.799917 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a52401-27c3-4577-916b-e7c7d902536a-openshift-service-ca\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.801660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.801635 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a52401-27c3-4577-916b-e7c7d902536a-maas-api-tls\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.809818 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.809794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mm5\" (UniqueName: \"kubernetes.io/projected/01a52401-27c3-4577-916b-e7c7d902536a-kube-api-access-z6mm5\") pod \"maas-api-5595cc7ddb-qxc92\" (UID: \"01a52401-27c3-4577-916b-e7c7d902536a\") " pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.926120 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.926084 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:44.990488 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:44.990408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:45.073379 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.073188 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5595cc7ddb-qxc92"] Jul 02 23:19:45.075827 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:19:45.075803 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a52401_27c3_4577_916b_e7c7d902536a.slice/crio-a1fd5382fdecffe11a1e11fa92c9417b65ca8163388c2236154a83656ac85c48 WatchSource:0}: Error finding container a1fd5382fdecffe11a1e11fa92c9417b65ca8163388c2236154a83656ac85c48: Status 404 returned error can't find the container with id a1fd5382fdecffe11a1e11fa92c9417b65ca8163388c2236154a83656ac85c48 Jul 02 23:19:45.081582 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.081558 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:45.095497 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.095464 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-96tbf"] Jul 02 23:19:45.098443 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.098425 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.101494 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.101472 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lw2wg\"" Jul 02 23:19:45.101616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.101518 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 23:19:45.109196 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.109172 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-96tbf"] Jul 02 23:19:45.118688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.118662 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-mgn7q"] Jul 02 23:19:45.121276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.121257 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.133031 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.133004 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-mgn7q"] Jul 02 23:19:45.202284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.202251 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-plugins-config-volume\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.202442 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.202308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.202442 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.202410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jc5q\" (UniqueName: \"kubernetes.io/projected/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-kube-api-access-9jc5q\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.202518 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.202439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbgsf\" (UniqueName: \"kubernetes.io/projected/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-kube-api-access-fbgsf\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.303565 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.303502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc5q\" (UniqueName: \"kubernetes.io/projected/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-kube-api-access-9jc5q\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.303774 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.303573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbgsf\" (UniqueName: \"kubernetes.io/projected/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-kube-api-access-fbgsf\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.303774 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.303667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-plugins-config-volume\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.303774 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.303720 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.304442 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.304417 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-plugins-config-volume\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.304567 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.304442 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-plugins-config-volume\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.311818 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.311793 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc5q\" (UniqueName: \"kubernetes.io/projected/e6e30c1c-8554-49e3-b7c9-3c4959a81bf8-kube-api-access-9jc5q\") pod \"payload-processing-58f86c7798-mgn7q\" (UID: \"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8\") " pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.311926 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.311830 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbgsf\" (UniqueName: \"kubernetes.io/projected/2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5-kube-api-access-fbgsf\") pod \"payload-pre-processing-7964457696-96tbf\" (UID: \"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5\") " pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.411610 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.411510 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:45.436702 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.436672 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:45.572003 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.571975 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-96tbf"] Jul 02 23:19:45.574377 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:19:45.574350 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa1f5a4_6de1_4f32_a9c2_2e070a9489f5.slice/crio-b03ec96e74ab4fcd3f848690e81407d658175441e9a26b03d5f31d18647c39b5 WatchSource:0}: Error finding container b03ec96e74ab4fcd3f848690e81407d658175441e9a26b03d5f31d18647c39b5: Status 404 returned error can't find the container with id b03ec96e74ab4fcd3f848690e81407d658175441e9a26b03d5f31d18647c39b5 Jul 02 23:19:45.599063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.598888 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-mgn7q"] Jul 02 23:19:45.601726 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:19:45.601700 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e30c1c_8554_49e3_b7c9_3c4959a81bf8.slice/crio-62ea2bceecc143c02333d48194601758c39d54d053aebd6807e1d1bc5e4adfc5 WatchSource:0}: Error finding container 62ea2bceecc143c02333d48194601758c39d54d053aebd6807e1d1bc5e4adfc5: Status 404 returned error can't find the container with id 62ea2bceecc143c02333d48194601758c39d54d053aebd6807e1d1bc5e4adfc5 Jul 02 23:19:45.799306 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.799228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" event={"ID":"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5","Type":"ContainerStarted","Data":"b03ec96e74ab4fcd3f848690e81407d658175441e9a26b03d5f31d18647c39b5"} Jul 02 23:19:45.800758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.800730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" event={"ID":"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8","Type":"ContainerStarted","Data":"62ea2bceecc143c02333d48194601758c39d54d053aebd6807e1d1bc5e4adfc5"} Jul 02 23:19:45.802488 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:45.802465 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5595cc7ddb-qxc92" event={"ID":"01a52401-27c3-4577-916b-e7c7d902536a","Type":"ContainerStarted","Data":"a1fd5382fdecffe11a1e11fa92c9417b65ca8163388c2236154a83656ac85c48"} Jul 02 23:19:46.695231 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:46.695199 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:47.815836 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.815792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5595cc7ddb-qxc92" event={"ID":"01a52401-27c3-4577-916b-e7c7d902536a","Type":"ContainerStarted","Data":"25f1386cfe55c21a7e22265cb53046995a5b47d274211b7168275fe0f2b97f9a"} Jul 02 23:19:47.816290 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.815905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:47.817395 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.817363 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" event={"ID":"2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5","Type":"ContainerStarted","Data":"4d11036c51ed4c59cdd26495383c455c75d221445897dab600aba2ce2eb1e00f"} Jul 02 23:19:47.817516 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.817422 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:19:47.818729 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.818708 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" event={"ID":"e6e30c1c-8554-49e3-b7c9-3c4959a81bf8","Type":"ContainerStarted","Data":"24633dba74a444c748a54bf85ea8293cf2d8763aac4aecb3d6bbf9fbce2f97b8"} Jul 02 23:19:47.818813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.818757 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:47.834946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.834897 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5595cc7ddb-qxc92" podStartSLOduration=1.847264442 podStartE2EDuration="3.834879983s" totalImagesPullingTime="1.987615541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:19:47.833617532 +0000 UTC m=+534.325809207" watchObservedRunningTime="2026-07-02 23:19:47.834879983 +0000 UTC m=+534.327071658" Jul 02 23:19:47.851037 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.850979 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" podStartSLOduration=1.3879733220000001 podStartE2EDuration="2.850964023s" totalImagesPullingTime="1.462990701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:19:47.849161414 +0000 UTC m=+534.341353090" watchObservedRunningTime="2026-07-02 23:19:47.850964023 +0000 UTC m=+534.343155701" Jul 02 23:19:47.867312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:47.867256 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" podStartSLOduration=1.374968818 podStartE2EDuration="2.867239531s" totalImagesPullingTime="1.492270713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:19:47.865757299 +0000 UTC m=+534.357948975" watchObservedRunningTime="2026-07-02 23:19:47.867239531 +0000 UTC m=+534.359431204" Jul 02 23:19:53.830405 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:53.830370 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5595cc7ddb-qxc92" Jul 02 23:19:55.274334 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:55.274304 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:55.373633 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:55.373604 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:19:58.826732 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:58.826700 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-mgn7q" Jul 02 23:19:58.827247 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:19:58.826804 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-96tbf" Jul 02 23:20:04.965430 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.965396 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj"] Jul 02 23:20:04.970595 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.970505 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:04.973468 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.973437 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8c2j9\"" Jul 02 23:20:04.973468 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.973455 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 23:20:04.973702 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.973455 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 02 23:20:04.973702 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.973453 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 23:20:04.977960 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:04.977932 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj"] Jul 02 23:20:05.002444 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.002405 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:20:05.098444 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.098646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098455 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.098646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fa1544-a047-4a8b-baf6-748b1be7138a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.098646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098609 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.098849 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098654 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/65fa1544-a047-4a8b-baf6-748b1be7138a-kube-api-access-m2788\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.098849 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.098727 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199416 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199416 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199574 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fa1544-a047-4a8b-baf6-748b1be7138a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199624 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199724 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199654 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/65fa1544-a047-4a8b-baf6-748b1be7138a-kube-api-access-m2788\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199874 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.199932 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199874 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.200011 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.199990 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.200112 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.200095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.201819 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.201800 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fa1544-a047-4a8b-baf6-748b1be7138a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.202047 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.202032 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fa1544-a047-4a8b-baf6-748b1be7138a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.209631 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.209606 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/65fa1544-a047-4a8b-baf6-748b1be7138a-kube-api-access-m2788\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj\" (UID: \"65fa1544-a047-4a8b-baf6-748b1be7138a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.290350 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.290310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:05.434674 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.434647 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj"] Jul 02 23:20:05.436192 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:20:05.436165 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fa1544_a047_4a8b_baf6_748b1be7138a.slice/crio-910d52cd6188fd5432ae61f22044132e6dc26bac664e35d18d047be2b73ac6be WatchSource:0}: Error finding container 910d52cd6188fd5432ae61f22044132e6dc26bac664e35d18d047be2b73ac6be: Status 404 returned error can't find the container with id 910d52cd6188fd5432ae61f22044132e6dc26bac664e35d18d047be2b73ac6be Jul 02 23:20:05.913230 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:05.913181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" event={"ID":"65fa1544-a047-4a8b-baf6-748b1be7138a","Type":"ContainerStarted","Data":"910d52cd6188fd5432ae61f22044132e6dc26bac664e35d18d047be2b73ac6be"} Jul 02 23:20:06.920060 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:06.920026 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" event={"ID":"65fa1544-a047-4a8b-baf6-748b1be7138a","Type":"ContainerStarted","Data":"a4b37888ea26994adc6a4f54626415ff6dcae68f13acf48c05dd8d6706f4e0b6"} Jul 02 23:20:06.942753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:06.942698 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" podStartSLOduration=1.952385805 podStartE2EDuration="2.942681836s" totalImagesPullingTime="990.296031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:20:06.941360846 +0000 UTC m=+553.433552532" watchObservedRunningTime="2026-07-02 23:20:06.942681836 +0000 UTC m=+553.434873510" Jul 02 23:20:15.291045 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:15.291002 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:15.291528 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:15.291081 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:15.303958 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:15.303929 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:15.977946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:15.977911 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj" Jul 02 23:20:16.967780 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:16.967737 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg"] Jul 02 23:20:16.972969 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:16.972934 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:16.974594 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:16.974571 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg"] Jul 02 23:20:16.976502 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:16.976482 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 02 23:20:16.982160 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:16.982136 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:20:17.003626 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003595 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.003887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003663 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.003887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003819 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.003887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003848 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpgb\" (UniqueName: \"kubernetes.io/projected/55daa922-c4c0-4830-9b64-86a33110ec86-kube-api-access-4vpgb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.003887 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55daa922-c4c0-4830-9b64-86a33110ec86-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.004082 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.003948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105253 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpgb\" (UniqueName: \"kubernetes.io/projected/55daa922-c4c0-4830-9b64-86a33110ec86-kube-api-access-4vpgb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55daa922-c4c0-4830-9b64-86a33110ec86-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105400 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105705 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105470 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105793 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105769 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105884 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105863 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.105991 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.105966 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.108297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.108265 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55daa922-c4c0-4830-9b64-86a33110ec86-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.108434 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.108416 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55daa922-c4c0-4830-9b64-86a33110ec86-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.116971 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.116944 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpgb\" (UniqueName: \"kubernetes.io/projected/55daa922-c4c0-4830-9b64-86a33110ec86-kube-api-access-4vpgb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jlmpg\" (UID: \"55daa922-c4c0-4830-9b64-86a33110ec86\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.284320 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.284276 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:17.427467 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.427441 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg"] Jul 02 23:20:17.430669 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:20:17.430640 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55daa922_c4c0_4830_9b64_86a33110ec86.slice/crio-370d5fe9d6eb980f654837f129ed4bc0758af22cc1094ee31260a0ff3407bada WatchSource:0}: Error finding container 370d5fe9d6eb980f654837f129ed4bc0758af22cc1094ee31260a0ff3407bada: Status 404 returned error can't find the container with id 370d5fe9d6eb980f654837f129ed4bc0758af22cc1094ee31260a0ff3407bada Jul 02 23:20:17.978844 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.978756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" event={"ID":"55daa922-c4c0-4830-9b64-86a33110ec86","Type":"ContainerStarted","Data":"d970846a6643b37e67154e349f7dd6b5c28594b365815c5698aa99d40e0f939d"} Jul 02 23:20:17.978844 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.978809 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" event={"ID":"55daa922-c4c0-4830-9b64-86a33110ec86","Type":"ContainerStarted","Data":"370d5fe9d6eb980f654837f129ed4bc0758af22cc1094ee31260a0ff3407bada"} Jul 02 23:20:17.999687 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:17.999630 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" podStartSLOduration=1.827723218 podStartE2EDuration="1.99961466s" totalImagesPullingTime="171.891442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:20:17.997364342 +0000 UTC m=+564.489556019" watchObservedRunningTime="2026-07-02 23:20:17.99961466 +0000 UTC m=+564.491806342" Jul 02 23:20:24.194044 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:24.193995 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:20:27.284400 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:27.284364 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:27.284831 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:27.284445 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:27.297383 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:27.297355 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:28.039641 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:28.039609 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jlmpg" Jul 02 23:20:33.467813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:33.467766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:20:59.791303 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:20:59.791263 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:06.881249 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:06.881182 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:29.960007 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:29.959918 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:30.065654 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:30.065618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:30.162391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:30.162347 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:56.141769 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:56.141736 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:21:56.241210 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:21:56.241180 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:22:09.136223 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:22:09.136193 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:22:30.497101 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:22:30.497065 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:22:30.594723 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:22:30.594693 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:22:31.609249 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:22:31.609204 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:22:31.707405 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:22:31.707369 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:08.366352 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:08.366307 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:29.799916 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:29.799876 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:41.750211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:41.750173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:41.853973 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:41.853927 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:23:41.854207 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:41.854173 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-fccbb966b-pchs7" podUID="20934018-c9d3-4698-9df3-6fea7a2fe626" containerName="manager" containerID="cri-o://02dfa20f36d2bc62fa606f32c090a672107a4117470530fac3e2335a88964548" gracePeriod=30 Jul 02 23:23:42.008046 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.007954 2561 generic.go:356] "Generic (PLEG): container finished" podID="20934018-c9d3-4698-9df3-6fea7a2fe626" containerID="02dfa20f36d2bc62fa606f32c090a672107a4117470530fac3e2335a88964548" exitCode=0 Jul 02 23:23:42.008046 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.008022 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-pchs7" event={"ID":"20934018-c9d3-4698-9df3-6fea7a2fe626","Type":"ContainerDied","Data":"02dfa20f36d2bc62fa606f32c090a672107a4117470530fac3e2335a88964548"} Jul 02 23:23:42.107423 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.107396 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:23:42.234964 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.234924 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert\") pod \"20934018-c9d3-4698-9df3-6fea7a2fe626\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " Jul 02 23:23:42.235327 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.235041 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74z6\" (UniqueName: \"kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6\") pod \"20934018-c9d3-4698-9df3-6fea7a2fe626\" (UID: \"20934018-c9d3-4698-9df3-6fea7a2fe626\") " Jul 02 23:23:42.237030 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.236997 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6" (OuterVolumeSpecName: "kube-api-access-s74z6") pod "20934018-c9d3-4698-9df3-6fea7a2fe626" (UID: "20934018-c9d3-4698-9df3-6fea7a2fe626"). InnerVolumeSpecName "kube-api-access-s74z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:23:42.237030 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.237020 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "20934018-c9d3-4698-9df3-6fea7a2fe626" (UID: "20934018-c9d3-4698-9df3-6fea7a2fe626"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:23:42.335991 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.335910 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74z6\" (UniqueName: \"kubernetes.io/projected/20934018-c9d3-4698-9df3-6fea7a2fe626-kube-api-access-s74z6\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:23:42.335991 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:42.335938 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/20934018-c9d3-4698-9df3-6fea7a2fe626-webhook-cert\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:23:43.013972 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.013942 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-pchs7" Jul 02 23:23:43.013972 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.013958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-pchs7" event={"ID":"20934018-c9d3-4698-9df3-6fea7a2fe626","Type":"ContainerDied","Data":"f9daf680019044bf98585e6ca5f2ff6b7bab10107fe2d9a56417336fee60dd38"} Jul 02 23:23:43.014481 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.014023 2561 scope.go:117] "RemoveContainer" containerID="02dfa20f36d2bc62fa606f32c090a672107a4117470530fac3e2335a88964548" Jul 02 23:23:43.036507 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.036478 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:23:43.040051 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.040028 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fccbb966b-pchs7"] Jul 02 23:23:43.671134 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.671095 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:23:43.788524 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.788488 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fccbb966b-7wgdk"] Jul 02 23:23:43.796354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.789564 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20934018-c9d3-4698-9df3-6fea7a2fe626" containerName="manager" Jul 02 23:23:43.796354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.789591 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20934018-c9d3-4698-9df3-6fea7a2fe626" containerName="manager" Jul 02 23:23:43.796354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.789854 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20934018-c9d3-4698-9df3-6fea7a2fe626" containerName="manager" Jul 02 23:23:43.796354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.795256 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:43.800044 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.800017 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d4hs2\"" Jul 02 23:23:43.800200 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.800037 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 23:23:43.800200 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.800045 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 23:23:43.803676 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.803493 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fccbb966b-7wgdk"] Jul 02 23:23:43.951905 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.951800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf42x\" (UniqueName: \"kubernetes.io/projected/c83788e0-52bc-479d-9c15-c05a61c38745-kube-api-access-mf42x\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:43.951905 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:43.951880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c83788e0-52bc-479d-9c15-c05a61c38745-webhook-cert\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.052977 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.052941 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c83788e0-52bc-479d-9c15-c05a61c38745-webhook-cert\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.053474 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.053032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf42x\" (UniqueName: \"kubernetes.io/projected/c83788e0-52bc-479d-9c15-c05a61c38745-kube-api-access-mf42x\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.055421 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.055396 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c83788e0-52bc-479d-9c15-c05a61c38745-webhook-cert\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.063179 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.063150 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf42x\" (UniqueName: \"kubernetes.io/projected/c83788e0-52bc-479d-9c15-c05a61c38745-kube-api-access-mf42x\") pod \"maas-controller-fccbb966b-7wgdk\" (UID: \"c83788e0-52bc-479d-9c15-c05a61c38745\") " pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.069986 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.069959 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20934018-c9d3-4698-9df3-6fea7a2fe626" path="/var/lib/kubelet/pods/20934018-c9d3-4698-9df3-6fea7a2fe626/volumes" Jul 02 23:23:44.107996 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.107945 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:44.249073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.249050 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fccbb966b-7wgdk"] Jul 02 23:23:44.251626 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:23:44.251604 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83788e0_52bc_479d_9c15_c05a61c38745.slice/crio-1c114bc8fa47f75dd37b8bc180891f24e9442bd739e6b8114aa3d496c60f2d7a WatchSource:0}: Error finding container 1c114bc8fa47f75dd37b8bc180891f24e9442bd739e6b8114aa3d496c60f2d7a: Status 404 returned error can't find the container with id 1c114bc8fa47f75dd37b8bc180891f24e9442bd739e6b8114aa3d496c60f2d7a Jul 02 23:23:44.253324 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:44.253303 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:23:45.027882 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:45.027839 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-7wgdk" event={"ID":"c83788e0-52bc-479d-9c15-c05a61c38745","Type":"ContainerStarted","Data":"312db4fe0ecdaae44426facd445e726adddf7c805e72ad2d9b158c3b51505605"} Jul 02 23:23:45.028063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:45.027899 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fccbb966b-7wgdk" event={"ID":"c83788e0-52bc-479d-9c15-c05a61c38745","Type":"ContainerStarted","Data":"1c114bc8fa47f75dd37b8bc180891f24e9442bd739e6b8114aa3d496c60f2d7a"} Jul 02 23:23:45.028063 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:45.027993 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:23:45.047979 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:45.047920 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fccbb966b-7wgdk" podStartSLOduration=1.595293847 podStartE2EDuration="2.047901798s" totalImagesPullingTime="452.607951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:23:45.045939501 +0000 UTC m=+771.538131177" watchObservedRunningTime="2026-07-02 23:23:45.047901798 +0000 UTC m=+771.540093470" Jul 02 23:23:56.039737 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:23:56.039699 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fccbb966b-7wgdk" Jul 02 23:24:21.348016 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:21.347924 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:21.451184 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:21.451146 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:21.552584 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:21.552523 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:27.283131 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:27.283098 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:27.398260 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:27.398228 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:27.488199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:27.488157 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:27.620749 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:27.620661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:33.313196 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:33.313153 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:24:33.384514 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:24:33.384481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:25:49.066754 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:25:49.066673 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:27:47.665309 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:27:47.665261 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:28:26.569822 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:28:26.569783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:00.148004 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.147968 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717250-d4wkp"] Jul 02 23:30:00.151502 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.151480 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:00.160841 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.160805 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717250-d4wkp"] Jul 02 23:30:00.317969 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.317924 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfkbk\" (UniqueName: \"kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk\") pod \"maas-api-key-cleanup-29717250-d4wkp\" (UID: \"20ab4f7e-25a0-4581-9861-41d39057648f\") " pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:00.418950 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.418844 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfkbk\" (UniqueName: \"kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk\") pod \"maas-api-key-cleanup-29717250-d4wkp\" (UID: \"20ab4f7e-25a0-4581-9861-41d39057648f\") " pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:00.427230 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.427194 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfkbk\" (UniqueName: \"kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk\") pod \"maas-api-key-cleanup-29717250-d4wkp\" (UID: \"20ab4f7e-25a0-4581-9861-41d39057648f\") " pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:00.463946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.463909 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:00.810609 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.810527 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717250-d4wkp"] Jul 02 23:30:00.812570 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:30:00.812526 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ab4f7e_25a0_4581_9861_41d39057648f.slice/crio-5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89 WatchSource:0}: Error finding container 5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89: Status 404 returned error can't find the container with id 5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89 Jul 02 23:30:00.814353 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.814334 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:30:00.901515 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:00.901482 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" event={"ID":"20ab4f7e-25a0-4581-9861-41d39057648f","Type":"ContainerStarted","Data":"5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89"} Jul 02 23:30:03.918447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:03.918415 2561 generic.go:356] "Generic (PLEG): container finished" podID="20ab4f7e-25a0-4581-9861-41d39057648f" containerID="bd367b5c7ebe6b0c99614b5db9019541bd4cc9bf6944a3fddca73c0e3471f611" exitCode=0 Jul 02 23:30:03.918820 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:03.918491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" event={"ID":"20ab4f7e-25a0-4581-9861-41d39057648f","Type":"ContainerDied","Data":"bd367b5c7ebe6b0c99614b5db9019541bd4cc9bf6944a3fddca73c0e3471f611"} Jul 02 23:30:05.059979 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.059953 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:05.169618 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.169575 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfkbk\" (UniqueName: \"kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk\") pod \"20ab4f7e-25a0-4581-9861-41d39057648f\" (UID: \"20ab4f7e-25a0-4581-9861-41d39057648f\") " Jul 02 23:30:05.171622 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.171597 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk" (OuterVolumeSpecName: "kube-api-access-zfkbk") pod "20ab4f7e-25a0-4581-9861-41d39057648f" (UID: "20ab4f7e-25a0-4581-9861-41d39057648f"). InnerVolumeSpecName "kube-api-access-zfkbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:30:05.271227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.271182 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfkbk\" (UniqueName: \"kubernetes.io/projected/20ab4f7e-25a0-4581-9861-41d39057648f-kube-api-access-zfkbk\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:30:05.928665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.928634 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" Jul 02 23:30:05.928665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.928651 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717250-d4wkp" event={"ID":"20ab4f7e-25a0-4581-9861-41d39057648f","Type":"ContainerDied","Data":"5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89"} Jul 02 23:30:05.929023 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:05.928693 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abc3f94413743ad2bace417f5e17f33bbd9328525c71e19085645d7c5d58e89" Jul 02 23:30:34.420069 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:34.419986 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:30:34.505752 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:30:34.505721 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:34:59.190451 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:34:59.190371 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:06.594079 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:06.594034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:15.790043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:15.790001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:19.911030 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:19.911000 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:22.744411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:22.744363 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:22.843498 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:22.843464 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:23.188350 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:23.188316 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:28.953776 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:28.953742 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:29.015354 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:29.015309 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:29.151073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:29.151037 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:29.212500 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:29.212411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:30.340373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.340338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:30.440460 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.440410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:30.609270 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.609166 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:30.609757 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.609738 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ab4f7e-25a0-4581-9861-41d39057648f" containerName="cleanup" Jul 02 23:35:30.609757 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.609759 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ab4f7e-25a0-4581-9861-41d39057648f" containerName="cleanup" Jul 02 23:35:30.609919 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.609895 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ab4f7e-25a0-4581-9861-41d39057648f" containerName="cleanup" Jul 02 23:35:30.613193 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.613171 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.617144 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.617123 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-fc5f7b5c-serving-cert\"" Jul 02 23:35:30.630894 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.630865 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:30.647513 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.647475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8f9s\" (UniqueName: \"kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.647726 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.647572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.647726 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.647624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.662221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.662189 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:30.748970 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.748939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.749144 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.748999 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8f9s\" (UniqueName: \"kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.749144 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:30.749115 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-fc5f7b5c-serving-cert: secret "maas-api-e2e-ait-fc5f7b5c-serving-cert" not found Jul 02 23:35:30.749222 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:30.749200 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls podName:85a3997c-14b4-4839-ae18-8b3165fd709f nodeName:}" failed. No retries permitted until 2026-07-02 23:35:31.24917782 +0000 UTC m=+1477.741369474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls") pod "maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" (UID: "85a3997c-14b4-4839-ae18-8b3165fd709f") : secret "maas-api-e2e-ait-fc5f7b5c-serving-cert" not found Jul 02 23:35:30.749284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.749120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.749926 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.749905 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:30.759994 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:30.759968 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8f9s\" (UniqueName: \"kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:31.127592 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:31.127556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:31.226790 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:31.226747 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:31.253791 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:31.253763 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:31.253963 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:31.253923 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-fc5f7b5c-serving-cert: secret "maas-api-e2e-ait-fc5f7b5c-serving-cert" not found Jul 02 23:35:31.254030 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:31.254016 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls podName:85a3997c-14b4-4839-ae18-8b3165fd709f nodeName:}" failed. No retries permitted until 2026-07-02 23:35:32.253993396 +0000 UTC m=+1478.746185070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls") pod "maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" (UID: "85a3997c-14b4-4839-ae18-8b3165fd709f") : secret "maas-api-e2e-ait-fc5f7b5c-serving-cert" not found Jul 02 23:35:31.326399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:31.326359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:32.264487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.264445 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:32.267386 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.267357 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:32.423989 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.423947 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:32.562950 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.562924 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:32.565102 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:35:32.565076 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a3997c_14b4_4839_ae18_8b3165fd709f.slice/crio-528550f97a560255b267f2381f028e67f14fa4f9ae577bd8c0c1ea45de61d0cf WatchSource:0}: Error finding container 528550f97a560255b267f2381f028e67f14fa4f9ae577bd8c0c1ea45de61d0cf: Status 404 returned error can't find the container with id 528550f97a560255b267f2381f028e67f14fa4f9ae577bd8c0c1ea45de61d0cf Jul 02 23:35:32.566459 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.566439 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:35:32.761385 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:32.761346 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:33.550472 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:33.550434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" event={"ID":"85a3997c-14b4-4839-ae18-8b3165fd709f","Type":"ContainerStarted","Data":"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409"} Jul 02 23:35:33.550910 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:33.550490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" event={"ID":"85a3997c-14b4-4839-ae18-8b3165fd709f","Type":"ContainerStarted","Data":"528550f97a560255b267f2381f028e67f14fa4f9ae577bd8c0c1ea45de61d0cf"} Jul 02 23:35:33.550910 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:33.550628 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:33.587292 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:33.587236 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" podStartSLOduration=3.160433141 podStartE2EDuration="3.587217129s" totalImagesPullingTime="426.783988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:35:33.586554745 +0000 UTC m=+1480.078746421" watchObservedRunningTime="2026-07-02 23:35:33.587217129 +0000 UTC m=+1480.079408832" Jul 02 23:35:35.465584 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.465528 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:35.701826 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.701792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:35.795584 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.795524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:35.873883 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.873849 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:35.874056 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.874028 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" podUID="85a3997c-14b4-4839-ae18-8b3165fd709f" containerName="maas-api" containerID="cri-o://fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409" gracePeriod=30 Jul 02 23:35:35.911227 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:35.911194 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:36.127297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.127266 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:36.205049 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.205017 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") pod \"85a3997c-14b4-4839-ae18-8b3165fd709f\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " Jul 02 23:35:36.205257 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.205093 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca\") pod \"85a3997c-14b4-4839-ae18-8b3165fd709f\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " Jul 02 23:35:36.205257 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.205189 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8f9s\" (UniqueName: \"kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s\") pod \"85a3997c-14b4-4839-ae18-8b3165fd709f\" (UID: \"85a3997c-14b4-4839-ae18-8b3165fd709f\") " Jul 02 23:35:36.205475 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.205446 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85a3997c-14b4-4839-ae18-8b3165fd709f" (UID: "85a3997c-14b4-4839-ae18-8b3165fd709f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:35:36.205642 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.205623 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a3997c-14b4-4839-ae18-8b3165fd709f-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:36.208002 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.207977 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85a3997c-14b4-4839-ae18-8b3165fd709f" (UID: "85a3997c-14b4-4839-ae18-8b3165fd709f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:35:36.208085 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.208014 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s" (OuterVolumeSpecName: "kube-api-access-j8f9s") pod "85a3997c-14b4-4839-ae18-8b3165fd709f" (UID: "85a3997c-14b4-4839-ae18-8b3165fd709f"). InnerVolumeSpecName "kube-api-access-j8f9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:35:36.306860 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.306818 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a3997c-14b4-4839-ae18-8b3165fd709f-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:36.306860 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.306851 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8f9s\" (UniqueName: \"kubernetes.io/projected/85a3997c-14b4-4839-ae18-8b3165fd709f-kube-api-access-j8f9s\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:36.567066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.567027 2561 generic.go:356] "Generic (PLEG): container finished" podID="85a3997c-14b4-4839-ae18-8b3165fd709f" containerID="fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409" exitCode=0 Jul 02 23:35:36.567499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.567081 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" Jul 02 23:35:36.567499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.567112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" event={"ID":"85a3997c-14b4-4839-ae18-8b3165fd709f","Type":"ContainerDied","Data":"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409"} Jul 02 23:35:36.567499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.567169 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv" event={"ID":"85a3997c-14b4-4839-ae18-8b3165fd709f","Type":"ContainerDied","Data":"528550f97a560255b267f2381f028e67f14fa4f9ae577bd8c0c1ea45de61d0cf"} Jul 02 23:35:36.567499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.567190 2561 scope.go:117] "RemoveContainer" containerID="fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409" Jul 02 23:35:36.577285 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.577266 2561 scope.go:117] "RemoveContainer" containerID="fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409" Jul 02 23:35:36.577607 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:36.577588 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409\": container with ID starting with fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409 not found: ID does not exist" containerID="fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409" Jul 02 23:35:36.577698 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.577616 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409"} err="failed to get container status \"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409\": rpc error: code = NotFound desc = could not find container \"fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409\": container with ID starting with fdf69826aa4874e4fb2f940bc61cbc5a7de5c9d5baabeded6138a0127058a409 not found: ID does not exist" Jul 02 23:35:36.590427 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.590398 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:36.594474 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:36.594442 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-fc5f7b5c-86764f5cd4-jrcjv"] Jul 02 23:35:37.256553 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:37.256504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:38.068688 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:38.068653 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a3997c-14b4-4839-ae18-8b3165fd709f" path="/var/lib/kubelet/pods/85a3997c-14b4-4839-ae18-8b3165fd709f/volumes" Jul 02 23:35:41.474846 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:41.474811 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:41.546674 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:41.546642 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:41.721744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:41.721712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:41.874447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:41.874409 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:42.955485 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:42.955455 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:43.036208 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.036172 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:43.037869 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.037842 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:43.038371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.038350 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a3997c-14b4-4839-ae18-8b3165fd709f" containerName="maas-api" Jul 02 23:35:43.038470 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.038375 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a3997c-14b4-4839-ae18-8b3165fd709f" containerName="maas-api" Jul 02 23:35:43.038527 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.038514 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a3997c-14b4-4839-ae18-8b3165fd709f" containerName="maas-api" Jul 02 23:35:43.043060 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.043036 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.046021 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.046001 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-88a77b91-serving-cert\"" Jul 02 23:35:43.052506 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.052482 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:43.111950 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.111909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:43.168127 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.168094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrg4\" (UniqueName: \"kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.168127 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.168133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.168386 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.168199 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.215435 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.215328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:43.269727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.269685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrg4\" (UniqueName: \"kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.269727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.269728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.269947 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.269766 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.270505 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.270481 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.272253 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.272222 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.278975 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.278947 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrg4\" (UniqueName: \"kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4\") pod \"maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.356170 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.356126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:43.496877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.496852 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:43.498929 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:35:43.498897 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ab0c63_418e_4ddc_8c0e_48b05d27a777.slice/crio-0e7b9ff152b97e7d67047bcfb49761dec8486aff0f7e497fa2a5341f95107d06 WatchSource:0}: Error finding container 0e7b9ff152b97e7d67047bcfb49761dec8486aff0f7e497fa2a5341f95107d06: Status 404 returned error can't find the container with id 0e7b9ff152b97e7d67047bcfb49761dec8486aff0f7e497fa2a5341f95107d06 Jul 02 23:35:43.519032 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.519002 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:43.608146 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.608112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" event={"ID":"31ab0c63-418e-4ddc-8c0e-48b05d27a777","Type":"ContainerStarted","Data":"0e7b9ff152b97e7d67047bcfb49761dec8486aff0f7e497fa2a5341f95107d06"} Jul 02 23:35:43.621966 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:43.621934 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:44.615129 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:44.615093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" event={"ID":"31ab0c63-418e-4ddc-8c0e-48b05d27a777","Type":"ContainerStarted","Data":"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf"} Jul 02 23:35:44.615564 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:44.615205 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:44.632304 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:44.632257 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" podStartSLOduration=1.16235805 podStartE2EDuration="1.632242094s" totalImagesPullingTime="469.884044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:35:44.630939121 +0000 UTC m=+1491.123130798" watchObservedRunningTime="2026-07-02 23:35:44.632242094 +0000 UTC m=+1491.124433772" Jul 02 23:35:45.131201 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:45.131143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:48.016162 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.016129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:48.238124 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.238092 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:48.238837 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.238314 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" podUID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" containerName="maas-api" containerID="cri-o://ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf" gracePeriod=30 Jul 02 23:35:48.261218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.261190 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:48.494806 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.494774 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:48.622081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.621986 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfrg4\" (UniqueName: \"kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4\") pod \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " Jul 02 23:35:48.622081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.622048 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls\") pod \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " Jul 02 23:35:48.622317 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.622090 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca\") pod \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\" (UID: \"31ab0c63-418e-4ddc-8c0e-48b05d27a777\") " Jul 02 23:35:48.622592 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.622564 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31ab0c63-418e-4ddc-8c0e-48b05d27a777" (UID: "31ab0c63-418e-4ddc-8c0e-48b05d27a777"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:35:48.624113 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.624090 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4" (OuterVolumeSpecName: "kube-api-access-lfrg4") pod "31ab0c63-418e-4ddc-8c0e-48b05d27a777" (UID: "31ab0c63-418e-4ddc-8c0e-48b05d27a777"). InnerVolumeSpecName "kube-api-access-lfrg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:35:48.624185 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.624106 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31ab0c63-418e-4ddc-8c0e-48b05d27a777" (UID: "31ab0c63-418e-4ddc-8c0e-48b05d27a777"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:35:48.636326 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.636293 2561 generic.go:356] "Generic (PLEG): container finished" podID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" containerID="ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf" exitCode=0 Jul 02 23:35:48.636440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.636345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" event={"ID":"31ab0c63-418e-4ddc-8c0e-48b05d27a777","Type":"ContainerDied","Data":"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf"} Jul 02 23:35:48.636440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.636353 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" Jul 02 23:35:48.636440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.636420 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw" event={"ID":"31ab0c63-418e-4ddc-8c0e-48b05d27a777","Type":"ContainerDied","Data":"0e7b9ff152b97e7d67047bcfb49761dec8486aff0f7e497fa2a5341f95107d06"} Jul 02 23:35:48.636564 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.636444 2561 scope.go:117] "RemoveContainer" containerID="ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf" Jul 02 23:35:48.646090 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.646070 2561 scope.go:117] "RemoveContainer" containerID="ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf" Jul 02 23:35:48.646395 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:35:48.646375 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf\": container with ID starting with ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf not found: ID does not exist" containerID="ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf" Jul 02 23:35:48.646457 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.646407 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf"} err="failed to get container status \"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf\": rpc error: code = NotFound desc = could not find container \"ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf\": container with ID starting with ce288889c6c6c5ebf7b2dcc9a9496838164f3934e0421fe51a9d727110b51ebf not found: ID does not exist" Jul 02 23:35:48.658122 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.658088 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:48.661619 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.661595 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-88a77b91-67d8bf8f74-zzbjw"] Jul 02 23:35:48.723744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.723716 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrg4\" (UniqueName: \"kubernetes.io/projected/31ab0c63-418e-4ddc-8c0e-48b05d27a777-kube-api-access-lfrg4\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:48.723744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.723741 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31ab0c63-418e-4ddc-8c0e-48b05d27a777-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:48.723744 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:48.723753 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ab0c63-418e-4ddc-8c0e-48b05d27a777-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:35:50.069852 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:50.069818 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" path="/var/lib/kubelet/pods/31ab0c63-418e-4ddc-8c0e-48b05d27a777/volumes" Jul 02 23:35:54.294208 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:54.294175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:54.394349 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:54.394300 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:35:55.915454 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:35:55.915418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.283177 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:00.283129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.379205 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:00.379175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.476992 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:00.476959 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:00.547687 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:00.547584 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.653821 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.653789 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.730758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.730717 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:01.731905 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.731875 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" containerName="maas-api" Jul 02 23:36:01.731905 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.731904 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" containerName="maas-api" Jul 02 23:36:01.732089 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.732044 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ab0c63-418e-4ddc-8c0e-48b05d27a777" containerName="maas-api" Jul 02 23:36:01.736819 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.736788 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:01.736933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.736915 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.738449 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.738409 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.739510 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.739488 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-2eb256fc-serving-cert\"" Jul 02 23:36:01.809876 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.809845 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:01.855387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.855348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.855624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.855429 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.855624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.855454 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.957006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.956919 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.957006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.956973 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.957006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.957000 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.957218 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:36:01.957097 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-2eb256fc-serving-cert: secret "maas-api-e2e-derive-2eb256fc-serving-cert" not found Jul 02 23:36:01.957218 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:36:01.957178 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls podName:720e166f-b397-47a6-b562-6a48672953d4 nodeName:}" failed. No retries permitted until 2026-07-02 23:36:02.457160771 +0000 UTC m=+1508.949352428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls") pod "maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" (UID: "720e166f-b397-47a6-b562-6a48672953d4") : secret "maas-api-e2e-derive-2eb256fc-serving-cert" not found Jul 02 23:36:01.957571 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.957554 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:01.965808 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:01.965784 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:02.049300 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.049235 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:02.280455 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.280423 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:02.381472 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.381438 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:02.462003 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.461969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:02.464371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.464348 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") pod \"maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:02.648394 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.648310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:02.796175 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:02.796111 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:03.516657 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:03.516623 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:03.717477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:03.717442 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" event={"ID":"720e166f-b397-47a6-b562-6a48672953d4","Type":"ContainerStarted","Data":"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3"} Jul 02 23:36:03.717657 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:03.717503 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" event={"ID":"720e166f-b397-47a6-b562-6a48672953d4","Type":"ContainerStarted","Data":"a8114c530fae76b06c7cca043b193b190eee1986ffc170a78908b553d982d889"} Jul 02 23:36:03.717657 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:03.717601 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:03.739899 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:03.739852 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" podStartSLOduration=2.379423275 podStartE2EDuration="2.739834078s" totalImagesPullingTime="360.410803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:36:03.738756773 +0000 UTC m=+1510.230948449" watchObservedRunningTime="2026-07-02 23:36:03.739834078 +0000 UTC m=+1510.232025756" Jul 02 23:36:06.100844 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.100810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:06.300680 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.300647 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:06.362605 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.362492 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:06.362814 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.362781 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" podUID="720e166f-b397-47a6-b562-6a48672953d4" containerName="maas-api" containerID="cri-o://3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3" gracePeriod=30 Jul 02 23:36:06.387611 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.387585 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:06.440284 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.440245 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:06.616401 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.616343 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:06.699081 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.699046 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f\") pod \"720e166f-b397-47a6-b562-6a48672953d4\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " Jul 02 23:36:06.699242 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.699089 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") pod \"720e166f-b397-47a6-b562-6a48672953d4\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " Jul 02 23:36:06.699242 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.699133 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca\") pod \"720e166f-b397-47a6-b562-6a48672953d4\" (UID: \"720e166f-b397-47a6-b562-6a48672953d4\") " Jul 02 23:36:06.699578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.699523 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "720e166f-b397-47a6-b562-6a48672953d4" (UID: "720e166f-b397-47a6-b562-6a48672953d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:36:06.701209 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.701187 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f" (OuterVolumeSpecName: "kube-api-access-j9j6f") pod "720e166f-b397-47a6-b562-6a48672953d4" (UID: "720e166f-b397-47a6-b562-6a48672953d4"). InnerVolumeSpecName "kube-api-access-j9j6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:36:06.701209 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.701193 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "720e166f-b397-47a6-b562-6a48672953d4" (UID: "720e166f-b397-47a6-b562-6a48672953d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:36:06.734068 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.734034 2561 generic.go:356] "Generic (PLEG): container finished" podID="720e166f-b397-47a6-b562-6a48672953d4" containerID="3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3" exitCode=0 Jul 02 23:36:06.734218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.734082 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" Jul 02 23:36:06.734218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.734123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" event={"ID":"720e166f-b397-47a6-b562-6a48672953d4","Type":"ContainerDied","Data":"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3"} Jul 02 23:36:06.734218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.734181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m" event={"ID":"720e166f-b397-47a6-b562-6a48672953d4","Type":"ContainerDied","Data":"a8114c530fae76b06c7cca043b193b190eee1986ffc170a78908b553d982d889"} Jul 02 23:36:06.734218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.734207 2561 scope.go:117] "RemoveContainer" containerID="3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3" Jul 02 23:36:06.747017 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.746997 2561 scope.go:117] "RemoveContainer" containerID="3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3" Jul 02 23:36:06.747271 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:36:06.747253 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3\": container with ID starting with 3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3 not found: ID does not exist" containerID="3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3" Jul 02 23:36:06.747331 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.747279 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3"} err="failed to get container status \"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3\": rpc error: code = NotFound desc = could not find container \"3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3\": container with ID starting with 3a85ba03bcce5dd5cc0de20382cc50791b091c582d71ac22117fe075b689eda3 not found: ID does not exist" Jul 02 23:36:06.756682 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.756661 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:06.760232 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.760210 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-2eb256fc-7f46969685-5mm7m"] Jul 02 23:36:06.800331 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.800302 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9j6f\" (UniqueName: \"kubernetes.io/projected/720e166f-b397-47a6-b562-6a48672953d4-kube-api-access-j9j6f\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:36:06.800331 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.800332 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/720e166f-b397-47a6-b562-6a48672953d4-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:36:06.800501 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:06.800343 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/720e166f-b397-47a6-b562-6a48672953d4-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:36:08.069397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:08.069364 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720e166f-b397-47a6-b562-6a48672953d4" path="/var/lib/kubelet/pods/720e166f-b397-47a6-b562-6a48672953d4/volumes" Jul 02 23:36:09.593046 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:09.593014 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:11.568898 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:11.568864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:12.340487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:12.340458 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:12.401040 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:12.400999 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:12.797777 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:12.797748 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:12.895881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:12.895843 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:12.997568 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:12.997509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:15.380517 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:15.380474 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:21.409243 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:21.409207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:21.509627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:21.509591 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:21.608713 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:21.608685 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:21.708981 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:21.708893 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:22.078050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:22.078019 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:48.030614 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:48.030579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:53.606590 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:53.606529 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:53.707398 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:53.707362 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:36:53.809773 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:36:53.809742 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:09.671359 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:09.671319 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:15.495686 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:15.495650 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:15.651199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:15.651164 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:15.784956 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:15.784873 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:26.237609 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:26.237573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:27.021376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:27.021342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:32.714959 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:32.714928 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:32.822278 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:32.822241 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:32.916159 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:32.916121 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:33.018557 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:33.018503 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:33.761239 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:33.761201 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:33.849082 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:33.849051 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:33.953162 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:33.953129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:35.513995 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:35.513958 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:41.519216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:41.519165 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:41.713387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:41.713355 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:42.223431 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:42.223395 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:47.906734 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:47.906640 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:47.981516 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:47.981485 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:48.138340 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:48.138308 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:48.454091 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:48.454059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:48.784040 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:48.784003 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:51.290752 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.290691 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:51.347250 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.347218 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:37:51.347754 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.347731 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720e166f-b397-47a6-b562-6a48672953d4" containerName="maas-api" Jul 02 23:37:51.347754 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.347755 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="720e166f-b397-47a6-b562-6a48672953d4" containerName="maas-api" Jul 02 23:37:51.347914 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.347879 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="720e166f-b397-47a6-b562-6a48672953d4" containerName="maas-api" Jul 02 23:37:51.351147 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.351115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.354150 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.354131 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-5c689209-serving-cert\"" Jul 02 23:37:51.361513 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.361475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:37:51.382853 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.382817 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:51.464469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.464438 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:51.518044 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.518006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jwb\" (UniqueName: \"kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.518216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.518049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.518216 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.518072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.569388 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.569313 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:51.619397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.619357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jwb\" (UniqueName: \"kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.619397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.619400 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.619709 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.619425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.620137 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.620112 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.621932 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.621915 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.630769 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.630743 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jwb\" (UniqueName: \"kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb\") pod \"maas-api-e2e-mt-5c689209-6758f5d966-995ns\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.663756 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.663725 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:51.803830 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.803806 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:37:51.939756 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:51.939670 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:52.044649 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:52.044616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:52.277594 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:52.277561 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" event={"ID":"f8060dcd-f228-485b-bfd1-9bbd71bfacc7","Type":"ContainerStarted","Data":"47ad5287e48cf7730b4b967005fbf0bbb7eda18d91256ef9efa08ed259a4b235"} Jul 02 23:37:53.284620 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:53.284581 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" event={"ID":"f8060dcd-f228-485b-bfd1-9bbd71bfacc7","Type":"ContainerStarted","Data":"6bc945261e4f70130bfe5b1a2b0eef2a308eae7f858db2a0f246eed753e743db"} Jul 02 23:37:53.285013 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:53.284649 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:37:53.302057 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:53.302000 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" podStartSLOduration=1.8763269660000002 podStartE2EDuration="2.30198118s" totalImagesPullingTime="425.654214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:37:53.301097793 +0000 UTC m=+1619.793289470" watchObservedRunningTime="2026-07-02 23:37:53.30198118 +0000 UTC m=+1619.794172857" Jul 02 23:37:55.781813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:55.781780 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:56.423938 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:56.423904 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:56.526510 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:56.526475 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:56.872378 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:56.872349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:37:59.295380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:37:59.295348 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:38:02.558163 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:02.558126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:02.619428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:02.619395 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:02.942477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:02.942393 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:03.191761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.191728 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:38:03.192013 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.191981 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" podUID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" containerName="maas-api" containerID="cri-o://6bc945261e4f70130bfe5b1a2b0eef2a308eae7f858db2a0f246eed753e743db" gracePeriod=30 Jul 02 23:38:03.222135 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.222110 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:03.340267 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.340230 2561 generic.go:356] "Generic (PLEG): container finished" podID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" containerID="6bc945261e4f70130bfe5b1a2b0eef2a308eae7f858db2a0f246eed753e743db" exitCode=0 Jul 02 23:38:03.340463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.340299 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" event={"ID":"f8060dcd-f228-485b-bfd1-9bbd71bfacc7","Type":"ContainerDied","Data":"6bc945261e4f70130bfe5b1a2b0eef2a308eae7f858db2a0f246eed753e743db"} Jul 02 23:38:03.341390 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.341371 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:03.443624 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.443595 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:38:03.540016 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.539984 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls\") pod \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " Jul 02 23:38:03.540214 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.540043 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45jwb\" (UniqueName: \"kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb\") pod \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " Jul 02 23:38:03.540214 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.540064 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca\") pod \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\" (UID: \"f8060dcd-f228-485b-bfd1-9bbd71bfacc7\") " Jul 02 23:38:03.540465 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.540440 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8060dcd-f228-485b-bfd1-9bbd71bfacc7" (UID: "f8060dcd-f228-485b-bfd1-9bbd71bfacc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:38:03.542179 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.542151 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb" (OuterVolumeSpecName: "kube-api-access-45jwb") pod "f8060dcd-f228-485b-bfd1-9bbd71bfacc7" (UID: "f8060dcd-f228-485b-bfd1-9bbd71bfacc7"). InnerVolumeSpecName "kube-api-access-45jwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:38:03.542272 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.542175 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8060dcd-f228-485b-bfd1-9bbd71bfacc7" (UID: "f8060dcd-f228-485b-bfd1-9bbd71bfacc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:38:03.640855 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.640812 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:38:03.640855 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.640848 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jwb\" (UniqueName: \"kubernetes.io/projected/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-kube-api-access-45jwb\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:38:03.640855 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:03.640861 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8060dcd-f228-485b-bfd1-9bbd71bfacc7-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:38:04.346411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:04.346323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" event={"ID":"f8060dcd-f228-485b-bfd1-9bbd71bfacc7","Type":"ContainerDied","Data":"47ad5287e48cf7730b4b967005fbf0bbb7eda18d91256ef9efa08ed259a4b235"} Jul 02 23:38:04.346411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:04.346354 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns" Jul 02 23:38:04.346411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:04.346385 2561 scope.go:117] "RemoveContainer" containerID="6bc945261e4f70130bfe5b1a2b0eef2a308eae7f858db2a0f246eed753e743db" Jul 02 23:38:04.388602 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:04.388573 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:38:04.407252 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:04.407211 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-5c689209-6758f5d966-995ns"] Jul 02 23:38:06.069143 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:06.069108 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" path="/var/lib/kubelet/pods/f8060dcd-f228-485b-bfd1-9bbd71bfacc7/volumes" Jul 02 23:38:09.150568 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.150515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.250968 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.250932 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.430793 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.430719 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.616169 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.616134 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.740668 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.740580 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.838119 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.838087 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:09.947517 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:09.947487 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:15.724725 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:15.724662 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:15.832204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:15.832173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:15.925298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:15.925258 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:16.069323 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:16.069261 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:21.782252 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:21.782212 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:21.883318 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:21.883284 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:28.178034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:28.177999 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:28.482912 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:28.482807 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:28.732042 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:28.731987 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:31.452140 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.452086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:31.586454 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.586420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:31.679324 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.679292 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:38:31.680005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.679987 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" containerName="maas-api" Jul 02 23:38:31.680056 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.680008 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" containerName="maas-api" Jul 02 23:38:31.680112 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.680101 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8060dcd-f228-485b-bfd1-9bbd71bfacc7" containerName="maas-api" Jul 02 23:38:31.683480 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.683458 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.686417 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.686393 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-ea34dc2c-serving-cert\"" Jul 02 23:38:31.693833 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.693475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:38:31.714594 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.714503 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:31.814120 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.814083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzlj\" (UniqueName: \"kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.814339 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.814149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.814339 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.814184 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.915510 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.915476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzlj\" (UniqueName: \"kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.915711 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.915560 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.915711 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.915598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.916333 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.916315 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.918005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.917985 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.923670 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.923653 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzlj\" (UniqueName: \"kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj\") pod \"maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:31.933568 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.933520 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:31.998727 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:31.998695 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:32.144092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:32.144060 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:38:32.145673 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:38:32.145647 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda693d0aa_ffdc_4779_b40b_cfcab4478957.slice/crio-8c65fb521eb0c4e0288d473b74f601665596883aa38fe56ef235519dd2d2eabe WatchSource:0}: Error finding container 8c65fb521eb0c4e0288d473b74f601665596883aa38fe56ef235519dd2d2eabe: Status 404 returned error can't find the container with id 8c65fb521eb0c4e0288d473b74f601665596883aa38fe56ef235519dd2d2eabe Jul 02 23:38:32.283746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:32.283666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:32.384123 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:32.384092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:32.486740 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:32.486708 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" event={"ID":"a693d0aa-ffdc-4779-b40b-cfcab4478957","Type":"ContainerStarted","Data":"8c65fb521eb0c4e0288d473b74f601665596883aa38fe56ef235519dd2d2eabe"} Jul 02 23:38:33.494988 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:33.494952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" event={"ID":"a693d0aa-ffdc-4779-b40b-cfcab4478957","Type":"ContainerStarted","Data":"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4"} Jul 02 23:38:33.495383 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:33.495109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:33.514332 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:33.514278 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" podStartSLOduration=2.194588498 podStartE2EDuration="2.514263128s" totalImagesPullingTime="319.67463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:38:33.511692019 +0000 UTC m=+1660.003883696" watchObservedRunningTime="2026-07-02 23:38:33.514263128 +0000 UTC m=+1660.006454802" Jul 02 23:38:35.931689 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:35.931650 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:38:39.505657 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:39.505625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:38:43.390832 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:38:43.390799 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:08.090007 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:08.089974 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:08.191427 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:08.191388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:18.474877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:18.474790 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:18.910413 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:18.910380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:19.009741 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.009715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:19.119412 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.119377 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:39:19.119658 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.119628 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" podUID="a693d0aa-ffdc-4779-b40b-cfcab4478957" containerName="maas-api" containerID="cri-o://d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4" gracePeriod=30 Jul 02 23:39:19.149548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.149509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:19.377151 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.377124 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:39:19.564302 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.564266 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls\") pod \"a693d0aa-ffdc-4779-b40b-cfcab4478957\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " Jul 02 23:39:19.564834 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.564353 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca\") pod \"a693d0aa-ffdc-4779-b40b-cfcab4478957\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " Jul 02 23:39:19.564834 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.564379 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzlj\" (UniqueName: \"kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj\") pod \"a693d0aa-ffdc-4779-b40b-cfcab4478957\" (UID: \"a693d0aa-ffdc-4779-b40b-cfcab4478957\") " Jul 02 23:39:19.564834 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.564789 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a693d0aa-ffdc-4779-b40b-cfcab4478957" (UID: "a693d0aa-ffdc-4779-b40b-cfcab4478957"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:39:19.566427 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.566403 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj" (OuterVolumeSpecName: "kube-api-access-4wzlj") pod "a693d0aa-ffdc-4779-b40b-cfcab4478957" (UID: "a693d0aa-ffdc-4779-b40b-cfcab4478957"). InnerVolumeSpecName "kube-api-access-4wzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:39:19.566487 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.566408 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a693d0aa-ffdc-4779-b40b-cfcab4478957" (UID: "a693d0aa-ffdc-4779-b40b-cfcab4478957"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:39:19.665735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.665696 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a693d0aa-ffdc-4779-b40b-cfcab4478957-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:39:19.665735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.665727 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a693d0aa-ffdc-4779-b40b-cfcab4478957-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:39:19.665735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.665740 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzlj\" (UniqueName: \"kubernetes.io/projected/a693d0aa-ffdc-4779-b40b-cfcab4478957-kube-api-access-4wzlj\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:39:19.726255 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.726221 2561 generic.go:356] "Generic (PLEG): container finished" podID="a693d0aa-ffdc-4779-b40b-cfcab4478957" containerID="d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4" exitCode=0 Jul 02 23:39:19.726428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.726270 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" Jul 02 23:39:19.726428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.726278 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" event={"ID":"a693d0aa-ffdc-4779-b40b-cfcab4478957","Type":"ContainerDied","Data":"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4"} Jul 02 23:39:19.726428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.726321 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p" event={"ID":"a693d0aa-ffdc-4779-b40b-cfcab4478957","Type":"ContainerDied","Data":"8c65fb521eb0c4e0288d473b74f601665596883aa38fe56ef235519dd2d2eabe"} Jul 02 23:39:19.726428 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.726344 2561 scope.go:117] "RemoveContainer" containerID="d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4" Jul 02 23:39:19.737835 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.737812 2561 scope.go:117] "RemoveContainer" containerID="d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4" Jul 02 23:39:19.738580 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:39:19.738551 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4\": container with ID starting with d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4 not found: ID does not exist" containerID="d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4" Jul 02 23:39:19.738682 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.738591 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4"} err="failed to get container status \"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4\": rpc error: code = NotFound desc = could not find container \"d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4\": container with ID starting with d83692f5611f8d349d565174ac5274d4c4d3a0bc8a024c94a94239793eee76e4 not found: ID does not exist" Jul 02 23:39:19.747923 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.747899 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:39:19.752057 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:19.752036 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-ea34dc2c-68d7c994b5-4628p"] Jul 02 23:39:20.068346 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:20.068307 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a693d0aa-ffdc-4779-b40b-cfcab4478957" path="/var/lib/kubelet/pods/a693d0aa-ffdc-4779-b40b-cfcab4478957/volumes" Jul 02 23:39:24.563828 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:24.563785 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:25.327376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:25.327320 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:39:29.589419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:39:29.589379 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:05.775005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:05.774963 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:07.120010 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:07.119968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:12.218578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:12.218528 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:12.319256 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:12.319219 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:12.568201 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:12.568163 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:12.751920 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:12.751886 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:13.797909 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:13.797880 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:13.898691 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:13.898661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:13.999815 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:13.999783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:14.697397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:14.697359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:14.797446 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:14.797415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:14.896610 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:14.896581 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:16.248929 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:16.248895 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:21.813939 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:21.813909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:21.882793 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:21.882754 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:22.482644 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:22.482609 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:28.247400 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:28.247364 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:28.416187 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:28.416148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:28.499127 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:28.499040 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:38.946524 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:38.946493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:40:39.269202 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:40:39.269169 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:00.225736 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:00.225701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:06.022076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:06.022034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:06.121852 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:06.121823 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:06.349762 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:06.349670 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:06.664761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:06.664667 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:06.897825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:06.897793 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.281652 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.281603 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.333621 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.333586 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:41:09.334245 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.334225 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a693d0aa-ffdc-4779-b40b-cfcab4478957" containerName="maas-api" Jul 02 23:41:09.334295 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.334250 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a693d0aa-ffdc-4779-b40b-cfcab4478957" containerName="maas-api" Jul 02 23:41:09.334398 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.334386 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a693d0aa-ffdc-4779-b40b-cfcab4478957" containerName="maas-api" Jul 02 23:41:09.338194 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.338169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.341131 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.341114 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-a4ec8d-serving-cert\"" Jul 02 23:41:09.347781 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.347502 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:41:09.363690 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.363662 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.411105 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.411066 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2jq\" (UniqueName: \"kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.411301 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.411142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.411301 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.411198 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.426190 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.426155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.512036 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.511999 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.512215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.512066 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.512215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.512123 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2jq\" (UniqueName: \"kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.512215 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:41:09.512164 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-a4ec8d-serving-cert: secret "maas-api-e2e-inf-a-a4ec8d-serving-cert" not found Jul 02 23:41:09.512319 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:41:09.512256 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls podName:7024e508-2790-4316-bbec-82f2d0d87e71 nodeName:}" failed. No retries permitted until 2026-07-02 23:41:10.01223427 +0000 UTC m=+1816.504425940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls") pod "maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" (UID: "7024e508-2790-4316-bbec-82f2d0d87e71") : secret "maas-api-e2e-inf-a-a4ec8d-serving-cert" not found Jul 02 23:41:09.512802 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.512784 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.522075 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.522044 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2jq\" (UniqueName: \"kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:09.823006 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.822971 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.924819 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.924785 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:09.989470 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:09.989443 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:10.016968 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:10.016932 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:10.019575 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:10.019523 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:10.251944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:10.251916 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:10.391963 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:10.391942 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:41:10.394506 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:41:10.394474 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7024e508_2790_4316_bbec_82f2d0d87e71.slice/crio-49567e6b31d2e18c2df4e1eddf2a29cbaef80bfbabd5693d377adab1a0131a66 WatchSource:0}: Error finding container 49567e6b31d2e18c2df4e1eddf2a29cbaef80bfbabd5693d377adab1a0131a66: Status 404 returned error can't find the container with id 49567e6b31d2e18c2df4e1eddf2a29cbaef80bfbabd5693d377adab1a0131a66 Jul 02 23:41:10.395976 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:10.395956 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 23:41:11.289618 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:11.289579 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" event={"ID":"7024e508-2790-4316-bbec-82f2d0d87e71","Type":"ContainerStarted","Data":"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7"} Jul 02 23:41:11.289789 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:11.289665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" event={"ID":"7024e508-2790-4316-bbec-82f2d0d87e71","Type":"ContainerStarted","Data":"49567e6b31d2e18c2df4e1eddf2a29cbaef80bfbabd5693d377adab1a0131a66"} Jul 02 23:41:11.289789 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:11.289690 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:11.306661 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:11.306610 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" podStartSLOduration=1.8654238520000002 podStartE2EDuration="2.306594691s" totalImagesPullingTime="441.170839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:41:11.305004934 +0000 UTC m=+1817.797196610" watchObservedRunningTime="2026-07-02 23:41:11.306594691 +0000 UTC m=+1817.798786366" Jul 02 23:41:13.755789 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:13.755735 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:14.112934 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:14.112819 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:14.405947 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:14.405841 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:14.708023 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:14.707931 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:17.257168 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.257114 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:17.300567 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.300519 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:41:17.354411 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.354370 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:41:17.359483 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.359447 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.359746 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.359709 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:17.362101 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.362081 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-e96520-serving-cert\"" Jul 02 23:41:17.371772 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.371746 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:41:17.435664 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.435610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:17.493911 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.493875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.494102 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.493931 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.494102 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.494048 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvzg\" (UniqueName: \"kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.594734 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.594648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.594734 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.594697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.594734 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.594736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvzg\" (UniqueName: \"kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.595017 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:41:17.594811 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-e96520-serving-cert: secret "maas-api-e2e-inf-b-e96520-serving-cert" not found Jul 02 23:41:17.595017 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:41:17.594899 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls podName:d2b6e94c-8416-490e-a997-64421e058a99 nodeName:}" failed. No retries permitted until 2026-07-02 23:41:18.094878477 +0000 UTC m=+1824.587070133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls") pod "maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" (UID: "d2b6e94c-8416-490e-a997-64421e058a99") : secret "maas-api-e2e-inf-b-e96520-serving-cert" not found Jul 02 23:41:17.595338 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.595318 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.603763 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.603741 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvzg\" (UniqueName: \"kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:17.699696 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:17.699658 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:18.000570 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.000515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:18.099782 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.099740 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:18.101629 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.101599 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:18.102453 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.102431 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") pod \"maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:18.272174 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.272081 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:18.418651 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:18.418625 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:41:18.420756 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:41:18.420724 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b6e94c_8416_490e_a997_64421e058a99.slice/crio-dee8992152eab774cda929b276689e531bd2ffe5d1808f53a7e12b2a952dacec WatchSource:0}: Error finding container dee8992152eab774cda929b276689e531bd2ffe5d1808f53a7e12b2a952dacec: Status 404 returned error can't find the container with id dee8992152eab774cda929b276689e531bd2ffe5d1808f53a7e12b2a952dacec Jul 02 23:41:19.332221 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:19.332161 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" event={"ID":"d2b6e94c-8416-490e-a997-64421e058a99","Type":"ContainerStarted","Data":"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203"} Jul 02 23:41:19.332924 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:19.332261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" event={"ID":"d2b6e94c-8416-490e-a997-64421e058a99","Type":"ContainerStarted","Data":"dee8992152eab774cda929b276689e531bd2ffe5d1808f53a7e12b2a952dacec"} Jul 02 23:41:19.332924 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:19.332409 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:19.351337 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:19.351289 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" podStartSLOduration=2.036269809 podStartE2EDuration="2.351274276s" totalImagesPullingTime="315.004467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:41:19.349454338 +0000 UTC m=+1825.841646014" watchObservedRunningTime="2026-07-02 23:41:19.351274276 +0000 UTC m=+1825.843465950" Jul 02 23:41:21.748263 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:21.748222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:22.068312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:22.068206 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:25.343729 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:25.343700 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:41:30.867291 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:30.867254 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:30.967710 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:30.967680 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:39.526445 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:39.526402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:39.620937 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:39.620903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:45.167120 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:45.167067 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:48.490749 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:48.490711 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:48.550212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:48.550179 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:41:59.611328 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:41:59.611288 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:25.247842 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:25.247758 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.226630 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.226599 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.381405 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.381370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.761014 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.760985 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.861876 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.861842 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.961093 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.961064 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:26.998485 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.998454 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:42:26.998722 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:26.998698 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" podUID="7024e508-2790-4316-bbec-82f2d0d87e71" containerName="maas-api" containerID="cri-o://745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7" gracePeriod=30 Jul 02 23:42:27.034246 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.034223 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:42:27.249806 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.249777 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:42:27.389189 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.389102 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw2jq\" (UniqueName: \"kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq\") pod \"7024e508-2790-4316-bbec-82f2d0d87e71\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " Jul 02 23:42:27.389189 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.389164 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca\") pod \"7024e508-2790-4316-bbec-82f2d0d87e71\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " Jul 02 23:42:27.389648 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.389205 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") pod \"7024e508-2790-4316-bbec-82f2d0d87e71\" (UID: \"7024e508-2790-4316-bbec-82f2d0d87e71\") " Jul 02 23:42:27.389648 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.389629 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7024e508-2790-4316-bbec-82f2d0d87e71" (UID: "7024e508-2790-4316-bbec-82f2d0d87e71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:42:27.391220 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.391197 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7024e508-2790-4316-bbec-82f2d0d87e71" (UID: "7024e508-2790-4316-bbec-82f2d0d87e71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:42:27.391294 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.391241 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq" (OuterVolumeSpecName: "kube-api-access-hw2jq") pod "7024e508-2790-4316-bbec-82f2d0d87e71" (UID: "7024e508-2790-4316-bbec-82f2d0d87e71"). InnerVolumeSpecName "kube-api-access-hw2jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:42:27.490276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.490240 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7024e508-2790-4316-bbec-82f2d0d87e71-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:42:27.490276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.490269 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw2jq\" (UniqueName: \"kubernetes.io/projected/7024e508-2790-4316-bbec-82f2d0d87e71-kube-api-access-hw2jq\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:42:27.490276 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.490281 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7024e508-2790-4316-bbec-82f2d0d87e71-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:42:27.689694 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.689608 2561 generic.go:356] "Generic (PLEG): container finished" podID="7024e508-2790-4316-bbec-82f2d0d87e71" containerID="745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7" exitCode=0 Jul 02 23:42:27.689694 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.689667 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" Jul 02 23:42:27.689877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.689689 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" event={"ID":"7024e508-2790-4316-bbec-82f2d0d87e71","Type":"ContainerDied","Data":"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7"} Jul 02 23:42:27.689877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.689756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n" event={"ID":"7024e508-2790-4316-bbec-82f2d0d87e71","Type":"ContainerDied","Data":"49567e6b31d2e18c2df4e1eddf2a29cbaef80bfbabd5693d377adab1a0131a66"} Jul 02 23:42:27.689877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.689779 2561 scope.go:117] "RemoveContainer" containerID="745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7" Jul 02 23:42:27.700716 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.700659 2561 scope.go:117] "RemoveContainer" containerID="745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7" Jul 02 23:42:27.701081 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:42:27.701041 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7\": container with ID starting with 745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7 not found: ID does not exist" containerID="745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7" Jul 02 23:42:27.701212 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.701086 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7"} err="failed to get container status \"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7\": rpc error: code = NotFound desc = could not find container \"745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7\": container with ID starting with 745e8d63b0b64a774494119e0bd2d0f9cdb1d475e74ee2c4cba410fd35f6dfa7 not found: ID does not exist" Jul 02 23:42:27.721418 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.721392 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:42:27.725830 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:27.725809 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-a4ec8d-5594d46495-bq78n"] Jul 02 23:42:28.068610 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:28.068577 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7024e508-2790-4316-bbec-82f2d0d87e71" path="/var/lib/kubelet/pods/7024e508-2790-4316-bbec-82f2d0d87e71/volumes" Jul 02 23:42:31.939596 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:42:31.939518 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:08.597486 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:08.597450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:08.884034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:08.883942 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:09.091869 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.091839 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:09.254026 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.253970 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:09.670928 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.670810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:09.771247 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.771187 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:09.898836 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.898810 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:43:09.899102 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.899069 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" podUID="d2b6e94c-8416-490e-a997-64421e058a99" containerName="maas-api" containerID="cri-o://f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203" gracePeriod=30 Jul 02 23:43:09.920650 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:09.920626 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:10.147390 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.147367 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:43:10.312477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.312385 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") pod \"d2b6e94c-8416-490e-a997-64421e058a99\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " Jul 02 23:43:10.312477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.312431 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntvzg\" (UniqueName: \"kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg\") pod \"d2b6e94c-8416-490e-a997-64421e058a99\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " Jul 02 23:43:10.312477 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.312462 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca\") pod \"d2b6e94c-8416-490e-a997-64421e058a99\" (UID: \"d2b6e94c-8416-490e-a997-64421e058a99\") " Jul 02 23:43:10.312926 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.312902 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2b6e94c-8416-490e-a997-64421e058a99" (UID: "d2b6e94c-8416-490e-a997-64421e058a99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:43:10.314691 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.314661 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2b6e94c-8416-490e-a997-64421e058a99" (UID: "d2b6e94c-8416-490e-a997-64421e058a99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:43:10.314691 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.314673 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg" (OuterVolumeSpecName: "kube-api-access-ntvzg") pod "d2b6e94c-8416-490e-a997-64421e058a99" (UID: "d2b6e94c-8416-490e-a997-64421e058a99"). InnerVolumeSpecName "kube-api-access-ntvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:43:10.413745 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.413712 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b6e94c-8416-490e-a997-64421e058a99-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:10.413745 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.413739 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntvzg\" (UniqueName: \"kubernetes.io/projected/d2b6e94c-8416-490e-a997-64421e058a99-kube-api-access-ntvzg\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:10.413745 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.413748 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6e94c-8416-490e-a997-64421e058a99-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:10.911366 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.911329 2561 generic.go:356] "Generic (PLEG): container finished" podID="d2b6e94c-8416-490e-a997-64421e058a99" containerID="f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203" exitCode=0 Jul 02 23:43:10.911881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.911373 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" Jul 02 23:43:10.911881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.911396 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" event={"ID":"d2b6e94c-8416-490e-a997-64421e058a99","Type":"ContainerDied","Data":"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203"} Jul 02 23:43:10.911881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.911438 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p" event={"ID":"d2b6e94c-8416-490e-a997-64421e058a99","Type":"ContainerDied","Data":"dee8992152eab774cda929b276689e531bd2ffe5d1808f53a7e12b2a952dacec"} Jul 02 23:43:10.911881 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.911462 2561 scope.go:117] "RemoveContainer" containerID="f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203" Jul 02 23:43:10.921098 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.921080 2561 scope.go:117] "RemoveContainer" containerID="f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203" Jul 02 23:43:10.921375 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:10.921353 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203\": container with ID starting with f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203 not found: ID does not exist" containerID="f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203" Jul 02 23:43:10.921433 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.921384 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203"} err="failed to get container status \"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203\": rpc error: code = NotFound desc = could not find container \"f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203\": container with ID starting with f73a00076f7597d124024bfa7dd6d39d2151c7ec0b9f439b52b287b53c224203 not found: ID does not exist" Jul 02 23:43:10.936380 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.936344 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:43:10.942387 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:10.942359 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-e96520-5f6cbc57bb-6d86p"] Jul 02 23:43:12.068874 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:12.068837 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b6e94c-8416-490e-a997-64421e058a99" path="/var/lib/kubelet/pods/d2b6e94c-8416-490e-a997-64421e058a99/volumes" Jul 02 23:43:15.451825 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:15.451794 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:15.553317 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:15.553249 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:15.727655 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:15.727507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:15.948116 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:15.948085 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:16.284733 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:16.284701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:16.582283 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:16.582172 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:16.681842 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:16.681810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:19.823198 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.823159 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:19.893684 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.893642 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:19.894375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894355 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b6e94c-8416-490e-a997-64421e058a99" containerName="maas-api" Jul 02 23:43:19.894436 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894380 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b6e94c-8416-490e-a997-64421e058a99" containerName="maas-api" Jul 02 23:43:19.894436 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894397 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7024e508-2790-4316-bbec-82f2d0d87e71" containerName="maas-api" Jul 02 23:43:19.894436 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894405 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7024e508-2790-4316-bbec-82f2d0d87e71" containerName="maas-api" Jul 02 23:43:19.894583 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894507 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7024e508-2790-4316-bbec-82f2d0d87e71" containerName="maas-api" Jul 02 23:43:19.894583 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.894522 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b6e94c-8416-490e-a997-64421e058a99" containerName="maas-api" Jul 02 23:43:19.899167 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.899141 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:19.902666 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.902635 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-5898bd-serving-cert\"" Jul 02 23:43:19.912638 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.912615 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:19.924200 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:19.924166 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:20.007096 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.007053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8h6\" (UniqueName: \"kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.007288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.007183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.007288 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.007219 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.021166 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.021135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:20.108309 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.108201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.108309 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.108247 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.108553 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.108345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8h6\" (UniqueName: \"kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.109000 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.108966 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.110741 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.110723 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.117105 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.117075 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8h6\" (UniqueName: \"kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6\") pod \"maas-api-e2e-api-a-5898bd-56fc5dd685-vz655\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.121209 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.121160 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:20.212607 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.212557 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.350834 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.350808 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:20.353058 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:43:20.353016 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4721009_b39d_4588_81b3_7845dd9a6a83.slice/crio-1923e49138862cb3845814236c2e659a0c3a7276c16ba140a1f43c21e4376fbc WatchSource:0}: Error finding container 1923e49138862cb3845814236c2e659a0c3a7276c16ba140a1f43c21e4376fbc: Status 404 returned error can't find the container with id 1923e49138862cb3845814236c2e659a0c3a7276c16ba140a1f43c21e4376fbc Jul 02 23:43:20.375403 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.375316 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:20.479580 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.479524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:20.969166 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.969130 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" event={"ID":"c4721009-b39d-4588-81b3-7845dd9a6a83","Type":"ContainerStarted","Data":"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4"} Jul 02 23:43:20.969604 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.969180 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" event={"ID":"c4721009-b39d-4588-81b3-7845dd9a6a83","Type":"ContainerStarted","Data":"1923e49138862cb3845814236c2e659a0c3a7276c16ba140a1f43c21e4376fbc"} Jul 02 23:43:20.969604 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.969255 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:20.987928 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:20.987859 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" podStartSLOduration=1.555617011 podStartE2EDuration="1.987834984s" totalImagesPullingTime="432.217973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:43:20.986675485 +0000 UTC m=+1947.478867160" watchObservedRunningTime="2026-07-02 23:43:20.987834984 +0000 UTC m=+1947.480026660" Jul 02 23:43:24.311647 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:24.311618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:24.745034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:24.744950 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:25.062169 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:25.062141 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:25.291318 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:25.291287 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:26.980646 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:26.980615 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:27.925340 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:27.925267 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:27.995747 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:27.995711 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:27.999391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:27.999368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.004892 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.004863 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-dff6aa-serving-cert\"" Jul 02 23:43:28.012830 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.012687 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:28.033726 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.033686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:28.083285 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.083248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8mg\" (UniqueName: \"kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.083457 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.083363 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.083457 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.083414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.131469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.131438 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:28.185043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.184957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.185043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.185009 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.185043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.185044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8mg\" (UniqueName: \"kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.185300 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:28.185109 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-dff6aa-serving-cert: secret "maas-api-e2e-api-b-dff6aa-serving-cert" not found Jul 02 23:43:28.185300 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:28.185187 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls podName:1246bcc7-fde6-4f2b-8999-043804a067a2 nodeName:}" failed. No retries permitted until 2026-07-02 23:43:28.685170196 +0000 UTC m=+1955.177361849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls") pod "maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" (UID: "1246bcc7-fde6-4f2b-8999-043804a067a2") : secret "maas-api-e2e-api-b-dff6aa-serving-cert" not found Jul 02 23:43:28.185691 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.185672 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.196726 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.196695 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8mg\" (UniqueName: \"kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.484950 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.484840 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:28.581050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.581021 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:28.690371 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:28.690334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:28.690596 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:28.690572 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-dff6aa-serving-cert: secret "maas-api-e2e-api-b-dff6aa-serving-cert" not found Jul 02 23:43:28.690689 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:28.690676 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls podName:1246bcc7-fde6-4f2b-8999-043804a067a2 nodeName:}" failed. No retries permitted until 2026-07-02 23:43:29.690654471 +0000 UTC m=+1956.182846137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls") pod "maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" (UID: "1246bcc7-fde6-4f2b-8999-043804a067a2") : secret "maas-api-e2e-api-b-dff6aa-serving-cert" not found Jul 02 23:43:29.354318 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:29.354268 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:29.699564 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:29.699444 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:29.701888 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:29.701865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:29.817167 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:29.817127 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:29.957215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:29.957162 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:29.960250 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:43:29.960215 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1246bcc7_fde6_4f2b_8999_043804a067a2.slice/crio-0d8b2e601089f57da87220d21d6a91dc8cb220d4562b7faacdfdc06cb21878ee WatchSource:0}: Error finding container 0d8b2e601089f57da87220d21d6a91dc8cb220d4562b7faacdfdc06cb21878ee: Status 404 returned error can't find the container with id 0d8b2e601089f57da87220d21d6a91dc8cb220d4562b7faacdfdc06cb21878ee Jul 02 23:43:30.021076 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:30.021040 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" event={"ID":"1246bcc7-fde6-4f2b-8999-043804a067a2","Type":"ContainerStarted","Data":"0d8b2e601089f57da87220d21d6a91dc8cb220d4562b7faacdfdc06cb21878ee"} Jul 02 23:43:31.028214 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:31.028175 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" event={"ID":"1246bcc7-fde6-4f2b-8999-043804a067a2","Type":"ContainerStarted","Data":"1f02eea73bd72a5c9dd39192588efa7953afbc3974b7d5824cb3a3c02d12a021"} Jul 02 23:43:31.028632 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:31.028278 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:31.046970 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:31.046912 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" podStartSLOduration=3.632219042 podStartE2EDuration="4.046895088s" totalImagesPullingTime="414.676046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:43:31.045235552 +0000 UTC m=+1957.537427255" watchObservedRunningTime="2026-07-02 23:43:31.046895088 +0000 UTC m=+1957.539086791" Jul 02 23:43:32.412720 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:32.412684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:37.039738 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:37.039709 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:40.408176 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:40.408134 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:40.776467 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:40.776434 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:40.876661 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:40.876631 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:40.976807 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:40.976773 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:41.032905 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.032832 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:41.033050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.033025 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" podUID="c4721009-b39d-4588-81b3-7845dd9a6a83" containerName="maas-api" containerID="cri-o://2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4" gracePeriod=30 Jul 02 23:43:41.055892 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.055865 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:41.294707 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.294638 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:41.421970 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.421936 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca\") pod \"c4721009-b39d-4588-81b3-7845dd9a6a83\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " Jul 02 23:43:41.422379 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.422001 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls\") pod \"c4721009-b39d-4588-81b3-7845dd9a6a83\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " Jul 02 23:43:41.422379 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.422033 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv8h6\" (UniqueName: \"kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6\") pod \"c4721009-b39d-4588-81b3-7845dd9a6a83\" (UID: \"c4721009-b39d-4588-81b3-7845dd9a6a83\") " Jul 02 23:43:41.422448 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.422381 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4721009-b39d-4588-81b3-7845dd9a6a83" (UID: "c4721009-b39d-4588-81b3-7845dd9a6a83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:43:41.424165 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.424140 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4721009-b39d-4588-81b3-7845dd9a6a83" (UID: "c4721009-b39d-4588-81b3-7845dd9a6a83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:43:41.424238 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.424202 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6" (OuterVolumeSpecName: "kube-api-access-pv8h6") pod "c4721009-b39d-4588-81b3-7845dd9a6a83" (UID: "c4721009-b39d-4588-81b3-7845dd9a6a83"). InnerVolumeSpecName "kube-api-access-pv8h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:43:41.522870 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.522832 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv8h6\" (UniqueName: \"kubernetes.io/projected/c4721009-b39d-4588-81b3-7845dd9a6a83-kube-api-access-pv8h6\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:41.522870 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.522862 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4721009-b39d-4588-81b3-7845dd9a6a83-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:41.522870 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:41.522877 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4721009-b39d-4588-81b3-7845dd9a6a83-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:42.089070 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.089033 2561 generic.go:356] "Generic (PLEG): container finished" podID="c4721009-b39d-4588-81b3-7845dd9a6a83" containerID="2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4" exitCode=0 Jul 02 23:43:42.089236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.089078 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" Jul 02 23:43:42.089236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.089100 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" event={"ID":"c4721009-b39d-4588-81b3-7845dd9a6a83","Type":"ContainerDied","Data":"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4"} Jul 02 23:43:42.089236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.089147 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655" event={"ID":"c4721009-b39d-4588-81b3-7845dd9a6a83","Type":"ContainerDied","Data":"1923e49138862cb3845814236c2e659a0c3a7276c16ba140a1f43c21e4376fbc"} Jul 02 23:43:42.089236 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.089174 2561 scope.go:117] "RemoveContainer" containerID="2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4" Jul 02 23:43:42.101050 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.101029 2561 scope.go:117] "RemoveContainer" containerID="2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4" Jul 02 23:43:42.101379 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:43:42.101357 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4\": container with ID starting with 2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4 not found: ID does not exist" containerID="2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4" Jul 02 23:43:42.101456 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.101392 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4"} err="failed to get container status \"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4\": rpc error: code = NotFound desc = could not find container \"2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4\": container with ID starting with 2a973e6b67d9cd919fbcf5b3c2f0f8751f47c0933c36a15c16d1f8d4d57629e4 not found: ID does not exist" Jul 02 23:43:42.109941 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.109904 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:42.114345 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:42.114313 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-5898bd-56fc5dd685-vz655"] Jul 02 23:43:44.068440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:44.068407 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4721009-b39d-4588-81b3-7845dd9a6a83" path="/var/lib/kubelet/pods/c4721009-b39d-4588-81b3-7845dd9a6a83/volumes" Jul 02 23:43:46.617251 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:46.617168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:46.732083 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:46.732012 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:46.907498 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:46.907399 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.126182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.126139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.282013 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.281981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.654197 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.654088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.755469 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.755432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.854818 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.854783 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:47.896175 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.896144 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:47.896400 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.896374 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" podUID="1246bcc7-fde6-4f2b-8999-043804a067a2" containerName="maas-api" containerID="cri-o://1f02eea73bd72a5c9dd39192588efa7953afbc3974b7d5824cb3a3c02d12a021" gracePeriod=30 Jul 02 23:43:47.955961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:47.955927 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:48.121951 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.121923 2561 generic.go:356] "Generic (PLEG): container finished" podID="1246bcc7-fde6-4f2b-8999-043804a067a2" containerID="1f02eea73bd72a5c9dd39192588efa7953afbc3974b7d5824cb3a3c02d12a021" exitCode=0 Jul 02 23:43:48.122124 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.122001 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" event={"ID":"1246bcc7-fde6-4f2b-8999-043804a067a2","Type":"ContainerDied","Data":"1f02eea73bd72a5c9dd39192588efa7953afbc3974b7d5824cb3a3c02d12a021"} Jul 02 23:43:48.148612 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.148593 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:48.294778 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.294741 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") pod \"1246bcc7-fde6-4f2b-8999-043804a067a2\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " Jul 02 23:43:48.294949 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.294798 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8mg\" (UniqueName: \"kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg\") pod \"1246bcc7-fde6-4f2b-8999-043804a067a2\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " Jul 02 23:43:48.294949 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.294854 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca\") pod \"1246bcc7-fde6-4f2b-8999-043804a067a2\" (UID: \"1246bcc7-fde6-4f2b-8999-043804a067a2\") " Jul 02 23:43:48.295297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.295274 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1246bcc7-fde6-4f2b-8999-043804a067a2" (UID: "1246bcc7-fde6-4f2b-8999-043804a067a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:43:48.297035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.297010 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1246bcc7-fde6-4f2b-8999-043804a067a2" (UID: "1246bcc7-fde6-4f2b-8999-043804a067a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:43:48.297096 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.297058 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg" (OuterVolumeSpecName: "kube-api-access-8w8mg") pod "1246bcc7-fde6-4f2b-8999-043804a067a2" (UID: "1246bcc7-fde6-4f2b-8999-043804a067a2"). InnerVolumeSpecName "kube-api-access-8w8mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:43:48.396557 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.396495 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1246bcc7-fde6-4f2b-8999-043804a067a2-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:48.396557 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.396563 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1246bcc7-fde6-4f2b-8999-043804a067a2-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:48.396776 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:48.396580 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8mg\" (UniqueName: \"kubernetes.io/projected/1246bcc7-fde6-4f2b-8999-043804a067a2-kube-api-access-8w8mg\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:43:49.128608 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:49.128566 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" event={"ID":"1246bcc7-fde6-4f2b-8999-043804a067a2","Type":"ContainerDied","Data":"0d8b2e601089f57da87220d21d6a91dc8cb220d4562b7faacdfdc06cb21878ee"} Jul 02 23:43:49.129073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:49.128639 2561 scope.go:117] "RemoveContainer" containerID="1f02eea73bd72a5c9dd39192588efa7953afbc3974b7d5824cb3a3c02d12a021" Jul 02 23:43:49.129073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:49.128579 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq" Jul 02 23:43:49.152278 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:49.152251 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:49.156196 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:49.156168 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-dff6aa-85d766b8f4-nf6jq"] Jul 02 23:43:50.068339 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:50.068303 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1246bcc7-fde6-4f2b-8999-043804a067a2" path="/var/lib/kubelet/pods/1246bcc7-fde6-4f2b-8999-043804a067a2/volumes" Jul 02 23:43:53.633912 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:53.633881 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:53.729497 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:53.729462 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:53.891436 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:53.891339 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:54.088431 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:54.088398 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:54.424170 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:54.424139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:54.765855 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:54.765800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:55.233835 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:55.233759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.066721 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.066680 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.115515 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.115482 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:43:58.116039 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116015 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1246bcc7-fde6-4f2b-8999-043804a067a2" containerName="maas-api" Jul 02 23:43:58.116182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116159 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1246bcc7-fde6-4f2b-8999-043804a067a2" containerName="maas-api" Jul 02 23:43:58.116256 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116210 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4721009-b39d-4588-81b3-7845dd9a6a83" containerName="maas-api" Jul 02 23:43:58.116256 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116220 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4721009-b39d-4588-81b3-7845dd9a6a83" containerName="maas-api" Jul 02 23:43:58.116375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116341 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4721009-b39d-4588-81b3-7845dd9a6a83" containerName="maas-api" Jul 02 23:43:58.116375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.116357 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1246bcc7-fde6-4f2b-8999-043804a067a2" containerName="maas-api" Jul 02 23:43:58.119471 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.119450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.122149 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.122127 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-aefe3e-serving-cert\"" Jul 02 23:43:58.130001 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.129971 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:43:58.145246 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.145211 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.191463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.191420 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.191463 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.191464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fbq\" (UniqueName: \"kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.191735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.191618 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.205961 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.205932 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.292182 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.292150 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.292378 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.292230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.292378 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.292264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fbq\" (UniqueName: \"kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.292901 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.292876 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.294618 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.294597 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.300745 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.300726 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fbq\" (UniqueName: \"kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq\") pod \"maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.306928 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.306905 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.431257 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.431223 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:58.576277 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.576246 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:43:58.578024 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:43:58.577996 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3653d6_7444_4c0b_9dfe_3f1590ea2da9.slice/crio-ec6b7f9ab6b9fa3c9a461f89eec623719c3779b1d332303fa2333b7a07221b95 WatchSource:0}: Error finding container ec6b7f9ab6b9fa3c9a461f89eec623719c3779b1d332303fa2333b7a07221b95: Status 404 returned error can't find the container with id ec6b7f9ab6b9fa3c9a461f89eec623719c3779b1d332303fa2333b7a07221b95 Jul 02 23:43:58.618587 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.618519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:58.716111 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:58.716018 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:43:59.182917 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:59.182884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" event={"ID":"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9","Type":"ContainerStarted","Data":"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b"} Jul 02 23:43:59.183313 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:59.182939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" event={"ID":"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9","Type":"ContainerStarted","Data":"ec6b7f9ab6b9fa3c9a461f89eec623719c3779b1d332303fa2333b7a07221b95"} Jul 02 23:43:59.183313 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:59.183050 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:43:59.203999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:43:59.203951 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" podStartSLOduration=0.809211007 podStartE2EDuration="1.203935681s" totalImagesPullingTime="394.724674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:43:59.201744261 +0000 UTC m=+1985.693935938" watchObservedRunningTime="2026-07-02 23:43:59.203935681 +0000 UTC m=+1985.696127355" Jul 02 23:44:02.535218 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:02.535185 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:02.902167 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:02.902077 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:03.199300 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:03.199208 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:03.439133 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:03.439099 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:05.193459 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:05.193429 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:44:06.140500 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.140446 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:06.239985 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.239952 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:06.322581 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.322520 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:44:06.326581 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.326527 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.329298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.329272 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-b07c26-serving-cert\"" Jul 02 23:44:06.341877 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.341851 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:44:06.368153 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.368122 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:06.481863 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.481757 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.482034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.481861 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npv6r\" (UniqueName: \"kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.482034 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.481929 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.582634 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.582595 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.582843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.582683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.582843 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.582750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npv6r\" (UniqueName: \"kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.582843 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:44:06.582765 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-b07c26-serving-cert: secret "maas-api-e2e-shared-b-b07c26-serving-cert" not found Jul 02 23:44:06.582979 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:44:06.582848 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls podName:a1f4b4b5-bec0-4098-a006-c8fe5fd5cece nodeName:}" failed. No retries permitted until 2026-07-02 23:44:07.082825144 +0000 UTC m=+1993.575016797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls") pod "maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" (UID: "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece") : secret "maas-api-e2e-shared-b-b07c26-serving-cert" not found Jul 02 23:44:06.583419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.583399 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.591445 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.591414 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npv6r\" (UniqueName: \"kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:06.685460 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.685424 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:06.795312 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.795262 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:06.894115 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:06.894084 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:07.089294 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:07.089180 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:07.091838 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:07.091805 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") pod \"maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:07.237735 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:07.237689 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:07.381781 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:07.381753 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:44:07.384028 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:44:07.383997 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f4b4b5_bec0_4098_a006_c8fe5fd5cece.slice/crio-cbb7ae962583e87d00183798a8ea588b54d61fd9b733896e1b83baa8ba0d87e4 WatchSource:0}: Error finding container cbb7ae962583e87d00183798a8ea588b54d61fd9b733896e1b83baa8ba0d87e4: Status 404 returned error can't find the container with id cbb7ae962583e87d00183798a8ea588b54d61fd9b733896e1b83baa8ba0d87e4 Jul 02 23:44:08.230468 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.230432 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" event={"ID":"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece","Type":"ContainerStarted","Data":"982b15b69de8d3c6d3a052939023a64707253d392f6c600f080202d08bf028ff"} Jul 02 23:44:08.230678 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.230488 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" event={"ID":"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece","Type":"ContainerStarted","Data":"cbb7ae962583e87d00183798a8ea588b54d61fd9b733896e1b83baa8ba0d87e4"} Jul 02 23:44:08.230678 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.230524 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:08.249789 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.249734 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" podStartSLOduration=1.9259365480000001 podStartE2EDuration="2.249714121s" totalImagesPullingTime="323.777573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:44:08.247882775 +0000 UTC m=+1994.740074449" watchObservedRunningTime="2026-07-02 23:44:08.249714121 +0000 UTC m=+1994.741905796" Jul 02 23:44:08.867125 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.867072 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:08.959325 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:08.959293 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:10.637238 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:10.637203 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:14.241408 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:14.241380 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:44:18.742565 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:18.740767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:20.000271 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:20.000221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:20.100382 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:20.100341 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:25.798633 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:25.798603 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:28.228878 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:28.228831 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:28.349173 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:28.349120 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:47.469614 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:47.469577 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:44:53.087733 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:44:53.087701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:45:00.150952 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.150905 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p"] Jul 02 23:45:00.155068 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.155045 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r"] Jul 02 23:45:00.155220 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.155176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:00.158492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.158473 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-qd9zc"] Jul 02 23:45:00.158633 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.158598 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:00.162215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.162188 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:00.162654 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.162598 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p"] Jul 02 23:45:00.167616 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.167589 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-qd9zc"] Jul 02 23:45:00.170662 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.170637 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r"] Jul 02 23:45:00.214391 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.214355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bxq\" (UniqueName: \"kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq\") pod \"maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p\" (UID: \"76c991f1-abd9-479a-a6b7-000a4f66a290\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:00.214602 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.214415 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b2r\" (UniqueName: \"kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r\") pod \"maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r\" (UID: \"b216b829-a069-41f2-8bd6-d666f0433833\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:00.214602 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.214490 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m299\" (UniqueName: \"kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299\") pod \"maas-api-key-cleanup-29717265-qd9zc\" (UID: \"320c9291-95c7-4707-86bf-0ca67624d518\") " pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:00.315946 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.315906 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m299\" (UniqueName: \"kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299\") pod \"maas-api-key-cleanup-29717265-qd9zc\" (UID: \"320c9291-95c7-4707-86bf-0ca67624d518\") " pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:00.316132 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.316014 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bxq\" (UniqueName: \"kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq\") pod \"maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p\" (UID: \"76c991f1-abd9-479a-a6b7-000a4f66a290\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:00.316132 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.316056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b2r\" (UniqueName: \"kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r\") pod \"maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r\" (UID: \"b216b829-a069-41f2-8bd6-d666f0433833\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:00.324833 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.324810 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bxq\" (UniqueName: \"kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq\") pod \"maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p\" (UID: \"76c991f1-abd9-479a-a6b7-000a4f66a290\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:00.325005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.324983 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m299\" (UniqueName: \"kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299\") pod \"maas-api-key-cleanup-29717265-qd9zc\" (UID: \"320c9291-95c7-4707-86bf-0ca67624d518\") " pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:00.325052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.325011 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b2r\" (UniqueName: \"kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r\") pod \"maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r\" (UID: \"b216b829-a069-41f2-8bd6-d666f0433833\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:00.470777 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.470673 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:00.479859 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.479834 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:00.486866 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.486841 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:00.635603 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.635574 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p"] Jul 02 23:45:00.641742 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:45:00.641701 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c991f1_abd9_479a_a6b7_000a4f66a290.slice/crio-463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52 WatchSource:0}: Error finding container 463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52: Status 404 returned error can't find the container with id 463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52 Jul 02 23:45:00.877790 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.877615 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r"] Jul 02 23:45:00.879972 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:45:00.879941 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320c9291_95c7_4707_86bf_0ca67624d518.slice/crio-d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec WatchSource:0}: Error finding container d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec: Status 404 returned error can't find the container with id d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec Jul 02 23:45:00.880211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:00.880189 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717265-qd9zc"] Jul 02 23:45:00.881129 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:45:00.881101 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb216b829_a069_41f2_8bd6_d666f0433833.slice/crio-10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b WatchSource:0}: Error finding container 10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b: Status 404 returned error can't find the container with id 10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b Jul 02 23:45:01.508871 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.508834 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" event={"ID":"320c9291-95c7-4707-86bf-0ca67624d518","Type":"ContainerStarted","Data":"a9f3a565aa97a8ae5c67c6d479ff2a3fe05a2f8a0eb33d13d4956d2c2d37e018"} Jul 02 23:45:01.509361 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.508923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" event={"ID":"320c9291-95c7-4707-86bf-0ca67624d518","Type":"ContainerStarted","Data":"d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec"} Jul 02 23:45:01.510499 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.510471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" event={"ID":"76c991f1-abd9-479a-a6b7-000a4f66a290","Type":"ContainerStarted","Data":"8361bac1c21ba5f101d871d2ff03adeff7ee2931c8f3387c8d056fb24b24e84b"} Jul 02 23:45:01.510639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.510518 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" event={"ID":"76c991f1-abd9-479a-a6b7-000a4f66a290","Type":"ContainerStarted","Data":"463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52"} Jul 02 23:45:01.511925 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.511903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" event={"ID":"b216b829-a069-41f2-8bd6-d666f0433833","Type":"ContainerStarted","Data":"5b516f45c39da46b4fe912cbe06e6df8ec04597716a789fb5fba1fee0f2ebb24"} Jul 02 23:45:01.512035 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.511938 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" event={"ID":"b216b829-a069-41f2-8bd6-d666f0433833","Type":"ContainerStarted","Data":"10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b"} Jul 02 23:45:01.534364 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.534305 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" podStartSLOduration=1.5342861559999998 podStartE2EDuration="1.534286156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:45:01.534119116 +0000 UTC m=+2048.026310792" watchObservedRunningTime="2026-07-02 23:45:01.534286156 +0000 UTC m=+2048.026477832" Jul 02 23:45:01.551113 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.551058 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" podStartSLOduration=1.5510414479999999 podStartE2EDuration="1.551041448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:45:01.550512356 +0000 UTC m=+2048.042704036" watchObservedRunningTime="2026-07-02 23:45:01.551041448 +0000 UTC m=+2048.043233122" Jul 02 23:45:01.566353 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:01.566304 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" podStartSLOduration=1.566290323 podStartE2EDuration="1.566290323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:45:01.565048082 +0000 UTC m=+2048.057239758" watchObservedRunningTime="2026-07-02 23:45:01.566290323 +0000 UTC m=+2048.058481997" Jul 02 23:45:02.518199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.518159 2561 generic.go:356] "Generic (PLEG): container finished" podID="76c991f1-abd9-479a-a6b7-000a4f66a290" containerID="8361bac1c21ba5f101d871d2ff03adeff7ee2931c8f3387c8d056fb24b24e84b" exitCode=0 Jul 02 23:45:02.518691 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.518247 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" event={"ID":"76c991f1-abd9-479a-a6b7-000a4f66a290","Type":"ContainerDied","Data":"8361bac1c21ba5f101d871d2ff03adeff7ee2931c8f3387c8d056fb24b24e84b"} Jul 02 23:45:02.519839 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.519812 2561 generic.go:356] "Generic (PLEG): container finished" podID="b216b829-a069-41f2-8bd6-d666f0433833" containerID="5b516f45c39da46b4fe912cbe06e6df8ec04597716a789fb5fba1fee0f2ebb24" exitCode=0 Jul 02 23:45:02.519962 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.519903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" event={"ID":"b216b829-a069-41f2-8bd6-d666f0433833","Type":"ContainerDied","Data":"5b516f45c39da46b4fe912cbe06e6df8ec04597716a789fb5fba1fee0f2ebb24"} Jul 02 23:45:02.521357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.521334 2561 generic.go:356] "Generic (PLEG): container finished" podID="320c9291-95c7-4707-86bf-0ca67624d518" containerID="a9f3a565aa97a8ae5c67c6d479ff2a3fe05a2f8a0eb33d13d4956d2c2d37e018" exitCode=0 Jul 02 23:45:02.521445 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:02.521412 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" event={"ID":"320c9291-95c7-4707-86bf-0ca67624d518","Type":"ContainerDied","Data":"a9f3a565aa97a8ae5c67c6d479ff2a3fe05a2f8a0eb33d13d4956d2c2d37e018"} Jul 02 23:45:03.665203 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.665175 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:03.713831 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.713805 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:03.726204 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.726182 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:03.750130 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.750095 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6b2r\" (UniqueName: \"kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r\") pod \"b216b829-a069-41f2-8bd6-d666f0433833\" (UID: \"b216b829-a069-41f2-8bd6-d666f0433833\") " Jul 02 23:45:03.750311 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.750220 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8bxq\" (UniqueName: \"kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq\") pod \"76c991f1-abd9-479a-a6b7-000a4f66a290\" (UID: \"76c991f1-abd9-479a-a6b7-000a4f66a290\") " Jul 02 23:45:03.750311 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.750283 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m299\" (UniqueName: \"kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299\") pod \"320c9291-95c7-4707-86bf-0ca67624d518\" (UID: \"320c9291-95c7-4707-86bf-0ca67624d518\") " Jul 02 23:45:03.752357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.752321 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r" (OuterVolumeSpecName: "kube-api-access-q6b2r") pod "b216b829-a069-41f2-8bd6-d666f0433833" (UID: "b216b829-a069-41f2-8bd6-d666f0433833"). InnerVolumeSpecName "kube-api-access-q6b2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:45:03.752668 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.752641 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299" (OuterVolumeSpecName: "kube-api-access-4m299") pod "320c9291-95c7-4707-86bf-0ca67624d518" (UID: "320c9291-95c7-4707-86bf-0ca67624d518"). InnerVolumeSpecName "kube-api-access-4m299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:45:03.752761 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.752724 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq" (OuterVolumeSpecName: "kube-api-access-h8bxq") pod "76c991f1-abd9-479a-a6b7-000a4f66a290" (UID: "76c991f1-abd9-479a-a6b7-000a4f66a290"). InnerVolumeSpecName "kube-api-access-h8bxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:45:03.851554 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.851454 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m299\" (UniqueName: \"kubernetes.io/projected/320c9291-95c7-4707-86bf-0ca67624d518-kube-api-access-4m299\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:45:03.851554 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.851480 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6b2r\" (UniqueName: \"kubernetes.io/projected/b216b829-a069-41f2-8bd6-d666f0433833-kube-api-access-q6b2r\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:45:03.851554 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:03.851490 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8bxq\" (UniqueName: \"kubernetes.io/projected/76c991f1-abd9-479a-a6b7-000a4f66a290-kube-api-access-h8bxq\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:45:04.532885 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.532855 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" Jul 02 23:45:04.532885 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.532878 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r" event={"ID":"b216b829-a069-41f2-8bd6-d666f0433833","Type":"ContainerDied","Data":"10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b"} Jul 02 23:45:04.533115 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.532923 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f4181ed68b550e0ffe90811f0f2ead189c60539fc46d7cab521ea34ee5a73b" Jul 02 23:45:04.534167 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.534143 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" Jul 02 23:45:04.534313 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.534190 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717265-qd9zc" event={"ID":"320c9291-95c7-4707-86bf-0ca67624d518","Type":"ContainerDied","Data":"d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec"} Jul 02 23:45:04.534313 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.534237 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ce09f58a0c03d926fbb3566f8a66ef59f14f4ee8fae0748cfc7535de3998ec" Jul 02 23:45:04.535628 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.535611 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" Jul 02 23:45:04.535729 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.535611 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p" event={"ID":"76c991f1-abd9-479a-a6b7-000a4f66a290","Type":"ContainerDied","Data":"463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52"} Jul 02 23:45:04.535729 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:45:04.535713 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="463770c1da6a2863f0e07f9ca469a17b4003e6f7a3a0f551275564732f666e52" Jul 02 23:46:48.343608 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:46:48.343471 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:46:49.468096 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:46:49.468053 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:46:56.595713 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:46:56.595659 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:47:13.902215 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:47:13.902155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:07.812644 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:07.812595 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:08.773820 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:08.773787 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:18.795447 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:18.795349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:42.102248 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:42.102207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:47.702173 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:47.702141 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:48:47.803578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:48:47.803547 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:09.534352 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:09.534315 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:09.634369 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:09.634330 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:21.143866 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:21.143825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:21.242298 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:21.242268 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:22.714748 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:22.714710 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:23.670957 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:23.670904 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:24.715776 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.715737 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:24.957094 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.957059 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:49:24.957399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.957345 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerName="maas-api" containerID="cri-o://c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b" gracePeriod=30 Jul 02 23:49:24.981683 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.981651 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p"] Jul 02 23:49:24.983660 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.983634 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:24.983925 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:24.983899 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-aefe3e-29717265-gq79p"] Jul 02 23:49:25.076889 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.076864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:25.214617 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.214595 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:49:25.302242 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.302213 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls\") pod \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " Jul 02 23:49:25.302440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.302298 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca\") pod \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " Jul 02 23:49:25.302440 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.302354 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2fbq\" (UniqueName: \"kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq\") pod \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\" (UID: \"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9\") " Jul 02 23:49:25.302758 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.302724 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" (UID: "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:49:25.304465 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.304442 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" (UID: "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:49:25.304644 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.304441 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq" (OuterVolumeSpecName: "kube-api-access-w2fbq") pod "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" (UID: "9f3653d6-7444-4c0b-9dfe-3f1590ea2da9"). InnerVolumeSpecName "kube-api-access-w2fbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:49:25.404103 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.404071 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:49:25.404103 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.404098 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:49:25.404103 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.404108 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fbq\" (UniqueName: \"kubernetes.io/projected/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9-kube-api-access-w2fbq\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:49:25.904064 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.904030 2561 generic.go:356] "Generic (PLEG): container finished" podID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerID="c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b" exitCode=0 Jul 02 23:49:25.904462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.904064 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" event={"ID":"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9","Type":"ContainerDied","Data":"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b"} Jul 02 23:49:25.904462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.904121 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" event={"ID":"9f3653d6-7444-4c0b-9dfe-3f1590ea2da9","Type":"ContainerDied","Data":"ec6b7f9ab6b9fa3c9a461f89eec623719c3779b1d332303fa2333b7a07221b95"} Jul 02 23:49:25.904462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.904143 2561 scope.go:117] "RemoveContainer" containerID="c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b" Jul 02 23:49:25.904462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.904075 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" Jul 02 23:49:25.914083 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.914064 2561 scope.go:117] "RemoveContainer" containerID="c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b" Jul 02 23:49:25.914310 ip-10-0-133-16 kubenswrapper[2561]: E0702 23:49:25.914290 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b\": container with ID starting with c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b not found: ID does not exist" containerID="c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b" Jul 02 23:49:25.914356 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.914334 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b"} err="failed to get container status \"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b\": rpc error: code = NotFound desc = could not find container \"c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b\": container with ID starting with c5fb89a8909bb9692a07de32a666cfd3c218ad06e4bcb07ad080e6d9460d1b7b not found: ID does not exist" Jul 02 23:49:25.926328 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.926296 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:49:25.929858 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:25.929839 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95"] Jul 02 23:49:26.068092 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:26.068058 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c991f1-abd9-479a-a6b7-000a4f66a290" path="/var/lib/kubelet/pods/76c991f1-abd9-479a-a6b7-000a4f66a290/volumes" Jul 02 23:49:26.068461 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:26.068447 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" path="/var/lib/kubelet/pods/9f3653d6-7444-4c0b-9dfe-3f1590ea2da9/volumes" Jul 02 23:49:28.189179 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:28.189126 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-a-aefe3e-7994cf47f9-lzw95" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.50:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 02 23:49:29.499937 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:29.499896 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:29.601501 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:29.601466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:49:30.158674 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:49:30.158643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:06.921181 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:06.921131 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:07.174990 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.174878 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:07.397324 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.397282 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:07.566703 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.566672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:07.912944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.912863 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:50:07.913117 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.913080 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" podUID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" containerName="maas-api" containerID="cri-o://982b15b69de8d3c6d3a052939023a64707253d392f6c600f080202d08bf028ff" gracePeriod=30 Jul 02 23:50:07.949265 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.949240 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r"] Jul 02 23:50:07.954311 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.954286 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-b07c26-29717265-rp62r"] Jul 02 23:50:07.954844 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:07.954823 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.019049 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.019019 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:08.076001 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.075967 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b216b829-a069-41f2-8bd6-d666f0433833" path="/var/lib/kubelet/pods/b216b829-a069-41f2-8bd6-d666f0433833/volumes" Jul 02 23:50:08.118043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.118010 2561 generic.go:356] "Generic (PLEG): container finished" podID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" containerID="982b15b69de8d3c6d3a052939023a64707253d392f6c600f080202d08bf028ff" exitCode=0 Jul 02 23:50:08.118043 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.118053 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" event={"ID":"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece","Type":"ContainerDied","Data":"982b15b69de8d3c6d3a052939023a64707253d392f6c600f080202d08bf028ff"} Jul 02 23:50:08.166362 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.166306 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:50:08.305373 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.305336 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npv6r\" (UniqueName: \"kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r\") pod \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " Jul 02 23:50:08.305586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.305391 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca\") pod \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " Jul 02 23:50:08.305586 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.305503 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") pod \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\" (UID: \"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece\") " Jul 02 23:50:08.305805 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.305772 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" (UID: "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 23:50:08.307495 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.307468 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r" (OuterVolumeSpecName: "kube-api-access-npv6r") pod "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" (UID: "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece"). InnerVolumeSpecName "kube-api-access-npv6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 23:50:08.307614 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.307474 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" (UID: "a1f4b4b5-bec0-4098-a006-c8fe5fd5cece"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 23:50:08.407169 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.407134 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-maas-api-tls\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:50:08.407169 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.407165 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npv6r\" (UniqueName: \"kubernetes.io/projected/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-kube-api-access-npv6r\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:50:08.407169 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:08.407176 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece-openshift-service-ca\") on node \"ip-10-0-133-16.ec2.internal\" DevicePath \"\"" Jul 02 23:50:09.124742 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:09.124707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" event={"ID":"a1f4b4b5-bec0-4098-a006-c8fe5fd5cece","Type":"ContainerDied","Data":"cbb7ae962583e87d00183798a8ea588b54d61fd9b733896e1b83baa8ba0d87e4"} Jul 02 23:50:09.124742 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:09.124724 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx" Jul 02 23:50:09.125286 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:09.124760 2561 scope.go:117] "RemoveContainer" containerID="982b15b69de8d3c6d3a052939023a64707253d392f6c600f080202d08bf028ff" Jul 02 23:50:09.165462 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:09.165437 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:50:09.181409 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:09.181379 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b07c26-ddfc4b6c9-2ztzx"] Jul 02 23:50:10.071492 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:10.071460 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" path="/var/lib/kubelet/pods/a1f4b4b5-bec0-4098-a006-c8fe5fd5cece/volumes" Jul 02 23:50:13.856005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:13.855974 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:13.923229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:13.923200 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.125015 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:14.124940 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:14.350046 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:14.350015 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:19.508699 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:19.508671 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5595cc7ddb-qxc92_01a52401-27c3-4577-916b-e7c7d902536a/maas-api/0.log" Jul 02 23:50:19.620455 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:19.620432 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717250-d4wkp_20ab4f7e-25a0-4581-9861-41d39057648f/cleanup/0.log" Jul 02 23:50:19.723548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:19.723513 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717265-qd9zc_320c9291-95c7-4707-86bf-0ca67624d518/cleanup/0.log" Jul 02 23:50:19.843337 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:19.843251 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-fccbb966b-7wgdk_c83788e0-52bc-479d-9c15-c05a61c38745/manager/0.log" Jul 02 23:50:20.300525 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:20.300493 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9978976-nqdrr_3f6a5972-1a24-457a-945d-98ae86d9a347/copy-manifests/0.log" Jul 02 23:50:20.313545 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:20.313514 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9978976-nqdrr_3f6a5972-1a24-457a-945d-98ae86d9a347/manager/0.log" Jul 02 23:50:20.540917 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:20.540889 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-nncqg_3406c55b-22af-46b8-a49d-d505611cf820/postgres/0.log" Jul 02 23:50:23.071485 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:23.071452 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vh6bh_49bd5357-62f1-44b9-a4f8-cf4c51e2eb27/discovery/0.log" Jul 02 23:50:23.529764 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:23.529737 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-96tbf_2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5/payload-pre-processing/0.log" Jul 02 23:50:23.642949 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:23.642927 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-mgn7q_e6e30c1c-8554-49e3-b7c9-3c4959a81bf8/payload-processing/0.log" Jul 02 23:50:24.077358 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:24.077326 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qgzkj_65fa1544-a047-4a8b-baf6-748b1be7138a/main/0.log" Jul 02 23:50:24.186950 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:24.186922 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jlmpg_55daa922-c4c0-4830-9b64-86a33110ec86/main/0.log" Jul 02 23:50:29.210933 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:29.210894 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:29.307670 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:29.307639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" limit=1024 Jul 02 23:50:32.140301 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:32.140273 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8tm84_a7cc66b4-608b-4631-827a-2c764e4e023a/global-pull-secret-syncer/0.log" Jul 02 23:50:32.197757 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:32.197726 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5qjns_248d33c7-27bc-469d-8bbe-53669184b131/konnectivity-agent/0.log" Jul 02 23:50:32.327993 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:32.327965 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-16.ec2.internal_b956c94c524f1281d984172ee6c41e32/haproxy/0.log" Jul 02 23:50:38.936612 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:38.936581 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/alertmanager/0.log" Jul 02 23:50:38.961109 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:38.961089 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/config-reloader/0.log" Jul 02 23:50:38.985376 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:38.985298 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/kube-rbac-proxy-web/0.log" Jul 02 23:50:39.009634 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.009612 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/kube-rbac-proxy/0.log" Jul 02 23:50:39.034971 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.034948 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/kube-rbac-proxy-metric/0.log" Jul 02 23:50:39.064621 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.064600 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/prom-label-proxy/0.log" Jul 02 23:50:39.090548 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.090511 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9a8d0b-e257-4acf-8089-f9d5d539ab40/init-config-reloader/0.log" Jul 02 23:50:39.168503 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.168481 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nhgpw_766e3b40-8634-4674-b049-ecb926555f2e/kube-state-metrics/0.log" Jul 02 23:50:39.192229 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.192179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nhgpw_766e3b40-8634-4674-b049-ecb926555f2e/kube-rbac-proxy-main/0.log" Jul 02 23:50:39.217575 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.217555 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-nhgpw_766e3b40-8634-4674-b049-ecb926555f2e/kube-rbac-proxy-self/0.log" Jul 02 23:50:39.253652 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.253635 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f6f5b5756-nmbl7_6c8bac24-890b-4b15-95ae-39d365f41490/metrics-server/0.log" Jul 02 23:50:39.282891 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.282871 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-kbbh9_867ab172-93ec-4123-a022-0037bdc18c40/monitoring-plugin/0.log" Jul 02 23:50:39.333397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.333360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5lncx_d75c877b-90a7-4fab-855e-9db2d266ce0f/node-exporter/0.log" Jul 02 23:50:39.368432 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.368411 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5lncx_d75c877b-90a7-4fab-855e-9db2d266ce0f/kube-rbac-proxy/0.log" Jul 02 23:50:39.398139 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.398122 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5lncx_d75c877b-90a7-4fab-855e-9db2d266ce0f/init-textfile/0.log" Jul 02 23:50:39.603768 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.603739 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-688kw_4d75c08d-69a4-4063-a0ee-0af0190b793a/kube-rbac-proxy-main/0.log" Jul 02 23:50:39.632232 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.632206 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-688kw_4d75c08d-69a4-4063-a0ee-0af0190b793a/kube-rbac-proxy-self/0.log" Jul 02 23:50:39.654930 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.654910 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-688kw_4d75c08d-69a4-4063-a0ee-0af0190b793a/openshift-state-metrics/0.log" Jul 02 23:50:39.722502 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.722484 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/prometheus/0.log" Jul 02 23:50:39.744417 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.744398 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/config-reloader/0.log" Jul 02 23:50:39.766651 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.766628 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/thanos-sidecar/0.log" Jul 02 23:50:39.789910 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.789889 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/kube-rbac-proxy-web/0.log" Jul 02 23:50:39.813008 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.812957 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/kube-rbac-proxy/0.log" Jul 02 23:50:39.835160 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.835142 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/kube-rbac-proxy-thanos/0.log" Jul 02 23:50:39.861192 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.861177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79c921ba-7a0d-4e2d-be19-7ee98042d786/init-config-reloader/0.log" Jul 02 23:50:39.893639 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.893618 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-d5shl_7bf0327a-1414-4caf-bf9f-d2beaa09d07c/prometheus-operator/0.log" Jul 02 23:50:39.936439 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.936422 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-d5shl_7bf0327a-1414-4caf-bf9f-d2beaa09d07c/kube-rbac-proxy/0.log" Jul 02 23:50:39.962868 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.962848 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-r8tc8_cdfe9b5e-941f-4755-a3d1-a658a18dd443/prometheus-operator-admission-webhook/0.log" Jul 02 23:50:39.997397 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:39.997380 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c6f7795bd-7kz8m_210fa60d-c919-43b4-9ad0-112bb981f4c7/telemeter-client/0.log" Jul 02 23:50:40.021224 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.021195 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c6f7795bd-7kz8m_210fa60d-c919-43b4-9ad0-112bb981f4c7/reload/0.log" Jul 02 23:50:40.045516 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.045495 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c6f7795bd-7kz8m_210fa60d-c919-43b4-9ad0-112bb981f4c7/kube-rbac-proxy/0.log" Jul 02 23:50:40.081757 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.081661 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/thanos-query/0.log" Jul 02 23:50:40.105986 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.105964 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/kube-rbac-proxy-web/0.log" Jul 02 23:50:40.128944 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.128913 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/kube-rbac-proxy/0.log" Jul 02 23:50:40.157372 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.157355 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/prom-label-proxy/0.log" Jul 02 23:50:40.181523 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.181490 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/kube-rbac-proxy-rules/0.log" Jul 02 23:50:40.203875 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:40.203856 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7477f7856-rtvgb_b547ba94-c72a-456e-9959-9c264027b6fc/kube-rbac-proxy-metrics/0.log" Jul 02 23:50:42.165779 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.165724 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c"] Jul 02 23:50:42.166319 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166299 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" containerName="maas-api" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166322 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" containerName="maas-api" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166334 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerName="maas-api" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166340 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerName="maas-api" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166348 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320c9291-95c7-4707-86bf-0ca67624d518" containerName="cleanup" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166354 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="320c9291-95c7-4707-86bf-0ca67624d518" containerName="cleanup" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166366 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b216b829-a069-41f2-8bd6-d666f0433833" containerName="cleanup" Jul 02 23:50:42.166375 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166372 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b216b829-a069-41f2-8bd6-d666f0433833" containerName="cleanup" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166389 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c991f1-abd9-479a-a6b7-000a4f66a290" containerName="cleanup" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166394 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c991f1-abd9-479a-a6b7-000a4f66a290" containerName="cleanup" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166453 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f4b4b5-bec0-4098-a006-c8fe5fd5cece" containerName="maas-api" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166464 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="320c9291-95c7-4707-86bf-0ca67624d518" containerName="cleanup" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166470 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b216b829-a069-41f2-8bd6-d666f0433833" containerName="cleanup" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166477 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3653d6-7444-4c0b-9dfe-3f1590ea2da9" containerName="maas-api" Jul 02 23:50:42.166627 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.166484 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c991f1-abd9-479a-a6b7-000a4f66a290" containerName="cleanup" Jul 02 23:50:42.169753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.169738 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.172446 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.172418 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d4ctc\"/\"openshift-service-ca.crt\"" Jul 02 23:50:42.172590 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.172419 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d4ctc\"/\"default-dockercfg-jvzp8\"" Jul 02 23:50:42.173640 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.173622 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d4ctc\"/\"kube-root-ca.crt\"" Jul 02 23:50:42.178170 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.178151 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c"] Jul 02 23:50:42.331873 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.331839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-podres\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.332066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.331917 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-proc\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.332066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.331936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqgn\" (UniqueName: \"kubernetes.io/projected/d944a160-2e2c-4860-bec0-a45d6e5693a1-kube-api-access-wwqgn\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.332066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.331976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-lib-modules\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.332066 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.332043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-sys\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.432988 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.432896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-proc\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.432988 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.432935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqgn\" (UniqueName: \"kubernetes.io/projected/d944a160-2e2c-4860-bec0-a45d6e5693a1-kube-api-access-wwqgn\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433036 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-proc\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433087 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-lib-modules\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-sys\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433211 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-podres\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433245 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-sys\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433321 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-lib-modules\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.433399 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.433349 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d944a160-2e2c-4860-bec0-a45d6e5693a1-podres\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.442005 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.441983 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqgn\" (UniqueName: \"kubernetes.io/projected/d944a160-2e2c-4860-bec0-a45d6e5693a1-kube-api-access-wwqgn\") pod \"perf-node-gather-daemonset-fk74c\" (UID: \"d944a160-2e2c-4860-bec0-a45d6e5693a1\") " pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.480739 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.480706 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:42.588572 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.588514 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bb9cc4588-n2xfj_3417fe17-cdbc-4d05-9239-97ff4acb3d67/console/0.log" Jul 02 23:50:42.611199 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:42.611172 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c"] Jul 02 23:50:42.613168 ip-10-0-133-16 kubenswrapper[2561]: W0702 23:50:42.613140 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd944a160_2e2c_4860_bec0_a45d6e5693a1.slice/crio-f7ea3eb491d4dc1fede00da41b9adc93bc5af6d3f154468235e825943d6acae6 WatchSource:0}: Error finding container f7ea3eb491d4dc1fede00da41b9adc93bc5af6d3f154468235e825943d6acae6: Status 404 returned error can't find the container with id f7ea3eb491d4dc1fede00da41b9adc93bc5af6d3f154468235e825943d6acae6 Jul 02 23:50:43.300232 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:43.300198 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" event={"ID":"d944a160-2e2c-4860-bec0-a45d6e5693a1","Type":"ContainerStarted","Data":"596057da3271ec5b3d84edcea7a614edd5eab9bb2c53f0a8fc3676f285ef958c"} Jul 02 23:50:43.300753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:43.300259 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" event={"ID":"d944a160-2e2c-4860-bec0-a45d6e5693a1","Type":"ContainerStarted","Data":"f7ea3eb491d4dc1fede00da41b9adc93bc5af6d3f154468235e825943d6acae6"} Jul 02 23:50:43.300753 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:43.300313 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:43.316578 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:43.316511 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" podStartSLOduration=1.316498483 podStartE2EDuration="1.316498483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 23:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 23:50:43.316479917 +0000 UTC m=+2389.808671593" watchObservedRunningTime="2026-07-02 23:50:43.316498483 +0000 UTC m=+2389.808690158" Jul 02 23:50:44.287656 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:44.287629 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qb68g_db3559d7-7419-46ed-b51a-d7ec3d69da97/dns/0.log" Jul 02 23:50:44.310419 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:44.310394 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qb68g_db3559d7-7419-46ed-b51a-d7ec3d69da97/kube-rbac-proxy/0.log" Jul 02 23:50:44.385114 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:44.385092 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vp285_b9cbbc63-2b9c-4acf-9b24-1b8fe53b2e13/dns-node-resolver/0.log" Jul 02 23:50:45.022273 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:45.022246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwnt8_8610b55d-ca08-4394-a36c-001cb102480b/node-ca/0.log" Jul 02 23:50:46.105972 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:46.105943 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vh6bh_49bd5357-62f1-44b9-a4f8-cf4c51e2eb27/discovery/0.log" Jul 02 23:50:46.251014 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:46.250984 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-96tbf_2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5/payload-pre-processing/0.log" Jul 02 23:50:46.298813 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:46.298789 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-mgn7q_e6e30c1c-8554-49e3-b7c9-3c4959a81bf8/payload-processing/0.log" Jul 02 23:50:47.047880 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:47.047851 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-szfsv_f9526752-62a6-4815-8d5e-0a523d822544/serve-healthcheck-canary/0.log" Jul 02 23:50:47.682500 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:47.682472 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rv29q_09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6/kube-rbac-proxy/0.log" Jul 02 23:50:47.706580 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:47.706555 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rv29q_09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6/exporter/0.log" Jul 02 23:50:47.728314 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:47.728294 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rv29q_09cc0ae0-cdf9-4859-b5ec-d0c0a923f0b6/extractor/0.log" Jul 02 23:50:49.315730 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:49.315706 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d4ctc/perf-node-gather-daemonset-fk74c" Jul 02 23:50:50.016425 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.016396 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5595cc7ddb-qxc92_01a52401-27c3-4577-916b-e7c7d902536a/maas-api/0.log" Jul 02 23:50:50.037931 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.037906 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717250-d4wkp_20ab4f7e-25a0-4581-9861-41d39057648f/cleanup/0.log" Jul 02 23:50:50.065114 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.065076 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717265-qd9zc_320c9291-95c7-4707-86bf-0ca67624d518/cleanup/0.log" Jul 02 23:50:50.173732 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.173697 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-fccbb966b-7wgdk_c83788e0-52bc-479d-9c15-c05a61c38745/manager/0.log" Jul 02 23:50:50.417781 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.417699 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9978976-nqdrr_3f6a5972-1a24-457a-945d-98ae86d9a347/manager/0.log" Jul 02 23:50:50.437665 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.437641 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc9978976-nqdrr_3f6a5972-1a24-457a-945d-98ae86d9a347/copy-manifests/0.log" Jul 02 23:50:50.516461 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:50.516434 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-nncqg_3406c55b-22af-46b8-a49d-d505611cf820/postgres/0.log" Jul 02 23:50:52.041415 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:52.041386 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9d6d7b94f-xqj59_bdc35939-6eb0-451f-b8e4-457c8eb85391/manager/0.log" Jul 02 23:50:52.074357 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:52.074336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dgwp9_780fdb6c-0c2b-4f3d-8f30-bcb079cd02dc/openshift-lws-operator/0.log" Jul 02 23:50:52.667571 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:52.667485 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vh6bh_49bd5357-62f1-44b9-a4f8-cf4c51e2eb27/discovery/0.log" Jul 02 23:50:52.768297 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:52.768271 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-96tbf_2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5/payload-pre-processing/0.log" Jul 02 23:50:52.805869 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:52.805841 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-mgn7q_e6e30c1c-8554-49e3-b7c9-3c4959a81bf8/payload-processing/0.log" Jul 02 23:50:54.109286 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:54.109212 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vh6bh_49bd5357-62f1-44b9-a4f8-cf4c51e2eb27/discovery/0.log" Jul 02 23:50:54.224570 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:54.224525 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-96tbf_2fa1f5a4-6de1-4f32-a9c2-2e070a9489f5/payload-pre-processing/0.log" Jul 02 23:50:54.262266 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:54.262241 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-mgn7q_e6e30c1c-8554-49e3-b7c9-3c4959a81bf8/payload-processing/0.log" Jul 02 23:50:58.588999 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.588973 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/kube-multus-additional-cni-plugins/0.log" Jul 02 23:50:58.612305 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.612283 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/egress-router-binary-copy/0.log" Jul 02 23:50:58.637403 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.637381 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/cni-plugins/0.log" Jul 02 23:50:58.660969 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.660907 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/bond-cni-plugin/0.log" Jul 02 23:50:58.684073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.684050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/routeoverride-cni/0.log" Jul 02 23:50:58.707982 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.707962 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/whereabouts-cni-bincopy/0.log" Jul 02 23:50:58.732073 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:58.732053 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pmmhc_1d12fc74-35e0-44d9-96e7-e3c99eb8cc29/whereabouts-cni/0.log" Jul 02 23:50:59.186052 ip-10-0-133-16 kubenswrapper[2561]: I0702 23:50:59.186025 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-twd6k_4241c9b5-8638-4e3f-a515-8a9862cdf8a2/kube-multus/0.log"