Jul 21 21:48:38.042899 ip-10-0-130-224 systemd[1]: Starting Kubernetes Kubelet... Jul 21 21:48:38.405700 ip-10-0-130-224 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:38.405700 ip-10-0-130-224 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 21:48:38.405700 ip-10-0-130-224 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:38.405700 ip-10-0-130-224 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 21:48:38.405700 ip-10-0-130-224 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:38.406411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.405765 2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 21:48:38.410577 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410555 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.410577 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410576 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.410577 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410580 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410584 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410589 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410592 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410595 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410598 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410601 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410604 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410607 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410610 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410613 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410615 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410618 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410623 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410625 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410628 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410631 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410634 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410637 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410639 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.410701 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410642 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410645 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410649 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410652 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410655 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410657 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410660 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410663 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410666 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410669 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410672 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410675 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410692 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410696 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410700 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410703 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410707 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410710 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410714 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410717 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.411214 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410720 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410722 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410725 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410728 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410731 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410733 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410737 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410739 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410742 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410745 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410747 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410750 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410753 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410756 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410759 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410764 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410768 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410772 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410775 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.411750 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410778 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410781 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410784 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410787 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410792 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410796 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410799 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410803 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410808 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410812 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410815 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410819 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410822 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410826 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410828 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410831 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410834 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410837 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410839 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.412237 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410842 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410845 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410847 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410850 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410853 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410855 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410858 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410861 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410864 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410867 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410870 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410873 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410876 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410879 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410882 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410885 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410889 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410891 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.410895 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412344 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.412738 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412353 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412357 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412360 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412364 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412367 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412371 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412375 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412378 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412381 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412384 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412386 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412389 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412392 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412395 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412398 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412401 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412404 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412407 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412410 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412412 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.413255 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412415 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412420 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412424 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412428 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412433 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412437 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412440 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412444 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412448 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412451 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412454 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412458 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412460 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412463 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412467 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412470 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412473 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412476 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412478 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.413824 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412482 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412485 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412488 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412490 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412493 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412496 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412498 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412501 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412504 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412506 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412509 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412512 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412514 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412517 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412519 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412527 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412530 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412533 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412535 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412538 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.414337 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412541 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412544 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412547 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412549 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412552 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412555 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412557 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412560 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412566 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412569 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412573 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412576 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412579 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412582 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412585 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412588 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412591 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412593 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412596 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.414875 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412599 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412602 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412604 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412607 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412610 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412613 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412615 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412618 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412623 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412626 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412628 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412631 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412634 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412641 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412644 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412647 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412650 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412652 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412655 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.412659 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.415386 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413160 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413167 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413171 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413174 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413178 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413180 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413184 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413187 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413190 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413193 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413196 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413199 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413203 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413207 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413210 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413213 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413216 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413219 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413222 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.415927 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413224 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413229 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413232 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413235 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413238 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413240 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413243 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413246 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413249 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413252 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413255 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413258 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413261 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413265 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413268 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413271 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413274 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413277 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413280 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413283 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.416418 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413286 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413289 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413292 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413295 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413298 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413301 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413303 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413306 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413309 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413312 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413314 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413318 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413322 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413327 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413330 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413332 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413335 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413338 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413341 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.416943 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413344 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413348 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413351 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413355 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413358 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413361 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413366 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413369 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413372 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413375 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413377 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413380 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413383 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413385 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413389 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413392 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413395 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413397 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413400 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.417431 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413403 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413405 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413408 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413411 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413413 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413416 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413419 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413423 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413426 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413429 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413431 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413435 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413438 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413440 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413443 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413446 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413449 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413452 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413454 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413458 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.417942 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413461 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.413464 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413541 2555 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413551 2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413558 2555 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413563 2555 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413568 2555 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413571 2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413577 2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413582 2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413586 2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413589 2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413593 2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413596 2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413600 2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413603 2555 flags.go:64] FLAG: --cgroup-root="" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413606 2555 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413609 2555 flags.go:64] FLAG: --client-ca-file="" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413612 2555 flags.go:64] FLAG: --cloud-provider="external" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413617 2555 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413622 2555 flags.go:64] FLAG: --cluster-domain="" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413625 2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413628 2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413632 2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 21:48:38.418461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413636 2555 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413640 2555 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413643 2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413647 2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413650 2555 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413653 2555 flags.go:64] FLAG: --contention-profiling="false" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413656 2555 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413661 2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413665 2555 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413668 2555 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413673 2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413676 2555 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413693 2555 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413697 2555 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413700 2555 flags.go:64] FLAG: --enable-server="true" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413703 2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413708 2555 flags.go:64] FLAG: --event-burst="100" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413713 2555 flags.go:64] FLAG: --event-qps="50" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413716 2555 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413720 2555 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413723 2555 flags.go:64] FLAG: --eviction-hard="" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413727 2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413730 2555 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413734 2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413737 2555 flags.go:64] FLAG: --eviction-soft="" Jul 21 21:48:38.419129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413740 2555 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413743 2555 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413748 2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413752 2555 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413755 2555 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413758 2555 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413761 2555 flags.go:64] FLAG: --feature-gates="" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413765 2555 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413768 2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413772 2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413776 2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413779 2555 flags.go:64] FLAG: --healthz-port="10248" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413782 2555 flags.go:64] FLAG: --help="false" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413786 2555 flags.go:64] FLAG: --hostname-override="ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413791 2555 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413794 2555 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413797 2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413801 2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413805 2555 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413809 2555 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413812 2555 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413815 2555 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413818 2555 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 21:48:38.419773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413821 2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413825 2555 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413829 2555 flags.go:64] FLAG: --kube-reserved="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413832 2555 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413835 2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413839 2555 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413842 2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413845 2555 flags.go:64] FLAG: --lock-file="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413848 2555 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413851 2555 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413854 2555 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413865 2555 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413869 2555 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413872 2555 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413875 2555 flags.go:64] FLAG: --logging-format="text" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413878 2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413881 2555 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413884 2555 flags.go:64] FLAG: --manifest-url="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413887 2555 flags.go:64] FLAG: --manifest-url-header="" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413892 2555 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413895 2555 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413900 2555 flags.go:64] FLAG: --max-pods="110" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413903 2555 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413907 2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413910 2555 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 21:48:38.420411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413913 2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413916 2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413919 2555 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413922 2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413930 2555 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413934 2555 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413937 2555 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413940 2555 flags.go:64] FLAG: --pod-cidr="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413943 2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413950 2555 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413953 2555 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413957 2555 flags.go:64] FLAG: --pods-per-core="0" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413961 2555 flags.go:64] FLAG: --port="10250" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413964 2555 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413967 2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0613bf2876547f08a" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413970 2555 flags.go:64] FLAG: --qos-reserved="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413974 2555 flags.go:64] FLAG: --read-only-port="10255" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413977 2555 flags.go:64] FLAG: --register-node="true" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413980 2555 flags.go:64] FLAG: --register-with-taints="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413984 2555 flags.go:64] FLAG: --registry-burst="10" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413987 2555 flags.go:64] FLAG: --registry-qps="5" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413990 2555 flags.go:64] FLAG: --reserved-cpus="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413993 2555 flags.go:64] FLAG: --reserved-memory="" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.413998 2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414001 2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 21:48:38.421089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414004 2555 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414007 2555 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414010 2555 flags.go:64] FLAG: --runonce="false" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414013 2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414017 2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414020 2555 flags.go:64] FLAG: --seccomp-default="false" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414023 2555 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414026 2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414029 2555 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414032 2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414036 2555 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414039 2555 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414042 2555 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414045 2555 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414048 2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414051 2555 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414054 2555 flags.go:64] FLAG: --system-cgroups="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414058 2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414064 2555 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414068 2555 flags.go:64] FLAG: --tls-cert-file="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414071 2555 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414075 2555 flags.go:64] FLAG: --tls-min-version="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414078 2555 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414081 2555 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414084 2555 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 21:48:38.421949 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414087 2555 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414090 2555 flags.go:64] FLAG: --v="2" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414094 2555 flags.go:64] FLAG: --version="false" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414099 2555 flags.go:64] FLAG: --vmodule="" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414103 2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414107 2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414229 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414245 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414249 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414252 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414255 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414257 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414261 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414265 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414269 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414272 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414275 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414278 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414281 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414284 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414287 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.422780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414290 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414293 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414295 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414299 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414303 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414307 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414310 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414313 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414316 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414318 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414321 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414324 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414326 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414329 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414331 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414334 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414337 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414340 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414343 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.423658 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414346 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414349 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414351 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414355 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414357 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414360 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414365 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414368 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414371 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414373 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414376 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414379 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414382 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414385 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414388 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414392 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414395 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414398 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414400 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414404 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.424200 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414406 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414409 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414412 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414415 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414418 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414421 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414424 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414426 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414429 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414433 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414435 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414438 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414441 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414443 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414446 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414450 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414453 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414456 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414460 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414462 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.424834 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414465 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414468 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414471 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414473 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414478 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414481 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414484 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414489 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414492 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414495 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414498 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414501 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414505 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414507 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414510 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414513 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414515 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414518 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414521 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414524 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.425358 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414526 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414529 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414531 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414534 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.414537 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.414544 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.424222 2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.424242 2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.424261 2555 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.424267 2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424331 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424337 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424340 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424344 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424347 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.425895 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424351 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424354 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424357 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424360 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424364 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424367 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424371 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424374 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424378 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424381 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424383 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424387 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424390 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424393 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424397 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424400 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424404 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424408 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424411 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.426296 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424414 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424417 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424420 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424424 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424427 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424430 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424432 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424435 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424438 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424441 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424444 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424447 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424450 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424453 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424455 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424458 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424461 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424463 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424466 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424469 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.426806 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424472 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424475 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424478 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424480 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424485 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424489 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424491 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424494 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424497 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424500 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424503 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424506 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424509 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424512 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424515 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424518 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424521 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424524 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424526 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424529 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.427327 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424532 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424535 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424538 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424541 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424543 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424546 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424549 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424552 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424554 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424557 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424560 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424563 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424566 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424569 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424572 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424576 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424579 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424582 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424585 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.427870 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424588 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424591 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424593 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424596 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424599 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424602 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424604 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424607 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424611 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424615 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424618 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424621 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424624 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424627 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424631 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.428371 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424634 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.424640 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424791 2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424798 2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424801 2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424804 2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424807 2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424810 2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424813 2555 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424816 2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424819 2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424822 2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424824 2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424828 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424832 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:38.428798 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424835 2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424839 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424842 2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424845 2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424848 2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424850 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424855 2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424859 2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424862 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424865 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424868 2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424872 2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424875 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424878 2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424881 2555 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424884 2555 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424887 2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424890 2555 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424893 2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424896 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:38.429178 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424899 2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424901 2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424904 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424907 2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424909 2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424912 2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424915 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424918 2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424921 2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424924 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424926 2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424930 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424933 2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424935 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424938 2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424941 2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424943 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424946 2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424949 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424952 2555 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:38.429716 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424954 2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424957 2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424959 2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424962 2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424965 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424968 2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424970 2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424973 2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424976 2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424979 2555 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424981 2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424984 2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424988 2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424991 2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424994 2555 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424997 2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.424999 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425002 2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425005 2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:38.430242 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425009 2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425013 2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425016 2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425019 2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425022 2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425025 2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425029 2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425032 2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425035 2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425037 2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425040 2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425043 2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425046 2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425048 2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425051 2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425055 2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425058 2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425061 2555 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425063 2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425066 2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:38.430780 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425069 2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425072 2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425075 2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425078 2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425080 2555 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425083 2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:38.425086 2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.425091 2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.425232 2555 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.429348 2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.430634 2555 server.go:1019] "Starting client certificate rotation" Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.430751 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:38.431315 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.430791 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:38.432967 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.432950 2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:38.435840 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.435789 2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:38.444648 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.444623 2555 log.go:25] "Validated CRI v1 runtime API" Jul 21 21:48:38.446790 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.446766 2555 log.go:25] "Validated CRI v1 image API" Jul 21 21:48:38.447728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.447709 2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 21:48:38.450259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.450232 2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ba0ff5da-5b17-4d2a-aec0-717a41c7e34c:/dev/nvme0n1p3 ca4be6e0-d5c9-4df2-a954-ae980ca98116:/dev/nvme0n1p4] Jul 21 21:48:38.450328 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.450266 2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 21:48:38.456464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.456320 2555 manager.go:216] Machine: {Timestamp:2026-07-21 21:48:38.455769964 +0000 UTC m=+0.306529788 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100934 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246fd4d3308bb8b0e8b763e8947bfb SystemUUID:ec246fd4-d330-8bb8-b0e8-b763e8947bfb BootID:7fa96968-6506-4ab1-a28b-f274a298e46b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:3e:44:6e:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:3e:44:6e:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:dc:00:33:55:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 21:48:38.456464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.456454 2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 21:48:38.456595 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.456549 2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 21:48:38.459097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.459053 2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 21:48:38.459263 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.459104 2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 21:48:38.459842 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.459814 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:38.460173 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.460160 2555 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 21:48:38.460213 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.460176 2555 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 21:48:38.460213 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.460185 2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:38.461040 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.461029 2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:38.462215 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.462201 2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 21:48:38.466138 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466118 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:38.466321 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466309 2555 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 21:48:38.466371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466349 2555 kubelet.go:486] "Attempting to sync node with API server" Jul 21 21:48:38.466371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466366 2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 21:48:38.466451 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466378 2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 21:48:38.466451 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466389 2555 kubelet.go:398] "Adding apiserver pod source" Jul 21 21:48:38.466451 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466402 2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 21:48:38.466914 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466896 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:38.466914 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.466916 2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:38.470588 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.470552 2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 21:48:38.472418 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.472403 2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 21:48:38.472478 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.472436 2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 21:48:38.474015 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474003 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 21:48:38.474051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474021 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 21:48:38.474051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474027 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 21:48:38.474051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474033 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 21:48:38.474051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474042 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 21:48:38.474051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474048 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474056 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474062 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474069 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474075 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474085 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 21:48:38.474217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474090 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 21:48:38.474738 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474725 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 21:48:38.474776 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.474740 2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 21:48:38.478402 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.478382 2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 21:48:38.478471 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.478380 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 21:48:38.478471 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.478418 2555 server.go:1268] "Started kubelet" Jul 21 21:48:38.478658 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.478573 2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 21:48:38.478731 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.478692 2555 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 21:48:38.478962 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.478930 2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 21:48:38.479775 ip-10-0-130-224 systemd[1]: Started Kubernetes Kubelet. Jul 21 21:48:38.479911 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.479870 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 21:48:38.480206 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.480184 2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 21:48:38.481717 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.481701 2555 server.go:310] "Adding debug handlers to kubelet server" Jul 21 21:48:38.483460 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.483429 2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 21:48:38.486881 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.486862 2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 21:48:38.486881 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.486880 2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:38.487036 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.486042 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-224.ec2.internal.18c46c6187042e1f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-224.ec2.internal,UID:ip-10-0-130-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-224.ec2.internal,},FirstTimestamp:2026-07-21 21:48:38.478384671 +0000 UTC m=+0.329144497,LastTimestamp:2026-07-21 21:48:38.478384671 +0000 UTC m=+0.329144497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-224.ec2.internal,}" Jul 21 21:48:38.487036 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.486957 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:38.487466 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.487444 2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 21:48:38.487466 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.487453 2555 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 21:48:38.487641 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.487478 2555 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 21:48:38.487724 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.487654 2555 reconstruct.go:97] "Volume reconstruction finished" Jul 21 21:48:38.487724 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.487663 2555 reconciler.go:29] "Reconciler: start to sync state" Jul 21 21:48:38.488382 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488356 2555 factory.go:55] Registering systemd factory Jul 21 21:48:38.488454 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488389 2555 factory.go:223] Registration of the systemd container factory successfully Jul 21 21:48:38.488532 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488514 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5n8z" Jul 21 21:48:38.488713 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488689 2555 factory.go:153] Registering CRI-O factory Jul 21 21:48:38.488713 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488710 2555 factory.go:223] Registration of the crio container factory successfully Jul 21 21:48:38.488847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488777 2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 21:48:38.488847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488807 2555 factory.go:103] Registering Raw factory Jul 21 21:48:38.488847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.488824 2555 manager.go:1196] Started watching for new ooms in manager Jul 21 21:48:38.489385 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.489362 2555 manager.go:349] Starting recovery of all containers Jul 21 21:48:38.492640 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.492602 2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 21:48:38.492860 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.492835 2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 21:48:38.497627 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.497597 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5n8z" Jul 21 21:48:38.502132 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.502110 2555 manager.go:354] Recovery completed Jul 21 21:48:38.506900 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.506882 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.510004 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.509981 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.510084 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510019 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.510084 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510030 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.510682 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510646 2555 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 21:48:38.510682 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510657 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.510682 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510663 2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 21:48:38.510886 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.510700 2555 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:38.511419 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.511403 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.511475 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.511432 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.511475 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.511443 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.512260 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.512187 2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-224.ec2.internal.18c46c6188e6a470 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-224.ec2.internal,UID:ip-10-0-130-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-224.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-224.ec2.internal,},FirstTimestamp:2026-07-21 21:48:38.510003312 +0000 UTC m=+0.360763140,LastTimestamp:2026-07-21 21:48:38.510003312 +0000 UTC m=+0.360763140,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-224.ec2.internal,}" Jul 21 21:48:38.512911 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.512898 2555 policy_none.go:49] "None policy: Start" Jul 21 21:48:38.512982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.512914 2555 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 21:48:38.512982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.512925 2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 21:48:38.515009 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.514997 2555 policy_none.go:47] "Start" Jul 21 21:48:38.555646 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.555424 2555 manager.go:337] "Starting Device Plugin manager" Jul 21 21:48:38.555816 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.555686 2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 21:48:38.555816 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.555700 2555 server.go:87] "Starting device plugin registration server" Jul 21 21:48:38.555966 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.555956 2555 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 21:48:38.556010 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.555969 2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 21:48:38.556103 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.556083 2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 21:48:38.556187 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.556173 2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 21:48:38.556187 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.556187 2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 21:48:38.556825 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.556804 2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 21:48:38.556907 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.556867 2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:38.656248 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.656173 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.656171 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.657244 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.657285 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.657301 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.657339 2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.658151 2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.658187 2555 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.658209 2555 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 21:48:38.658715 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.658246 2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 21:48:38.660184 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.660166 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:38.664312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.664295 2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.664394 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.664320 2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-224.ec2.internal\": node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:38.713055 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.713004 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:38.759130 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.759094 2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal"] Jul 21 21:48:38.759234 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.759182 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.760288 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.760268 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.760408 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.760306 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.760408 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.760321 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.762711 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.762698 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.762775 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.762757 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.762831 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.762786 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.763647 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.763629 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.763713 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.763658 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.763713 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.763670 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.764212 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.764196 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.764263 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.764225 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.764263 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.764235 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.765976 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.765960 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.766060 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.765991 2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:38.766735 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.766715 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:38.766806 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.766748 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:38.766806 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.766758 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:38.787903 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.787874 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-224.ec2.internal\" not found" node="ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.788781 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.788758 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.788855 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.788796 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3ed946aa6fec2dea4a6cd621aaa10c4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-224.ec2.internal\" (UID: \"b3ed946aa6fec2dea4a6cd621aaa10c4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.788855 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.788827 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.792944 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.792924 2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-224.ec2.internal\" not found" node="ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.813160 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.813124 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:38.889103 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889070 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3ed946aa6fec2dea4a6cd621aaa10c4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-224.ec2.internal\" (UID: \"b3ed946aa6fec2dea4a6cd621aaa10c4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.889259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889113 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.889259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889138 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.889259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889171 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.889259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889180 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d007f4d04ced0c10ebffd3e44178333-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal\" (UID: \"0d007f4d04ced0c10ebffd3e44178333\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.889259 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:38.889169 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3ed946aa6fec2dea4a6cd621aaa10c4-config\") pod \"kube-apiserver-proxy-ip-10-0-130-224.ec2.internal\" (UID: \"b3ed946aa6fec2dea4a6cd621aaa10c4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:38.914832 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:38.914745 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.015991 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.015956 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.093073 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.093046 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:39.095482 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.095456 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:39.116135 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.116116 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.217591 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.217486 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.316466 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.316441 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:39.318547 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.318528 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.420221 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.420185 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.429882 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.429851 2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 21:48:39.430061 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.430041 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:48:39.430123 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.430069 2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:48:39.504559 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.504484 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 21:43:38 +0000 UTC" deadline="2028-04-18 10:39:19.681823954 +0000 UTC" Jul 21 21:48:39.504559 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.504521 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15276h50m40.177306413s" Jul 21 21:48:39.521483 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.520972 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.621421 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.621381 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.649573 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:39.649539 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d007f4d04ced0c10ebffd3e44178333.slice/crio-95545d65b07310a2226f8d8fff848abd004dc8fc927e7ef9c14d43590b784450 WatchSource:0}: Error finding container 95545d65b07310a2226f8d8fff848abd004dc8fc927e7ef9c14d43590b784450: Status 404 returned error can't find the container with id 95545d65b07310a2226f8d8fff848abd004dc8fc927e7ef9c14d43590b784450 Jul 21 21:48:39.650129 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:39.650110 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ed946aa6fec2dea4a6cd621aaa10c4.slice/crio-3bd89a2e73cc40476778635c608591bcd80aed88e5b4a14d1b2ca1da46518ee6 WatchSource:0}: Error finding container 3bd89a2e73cc40476778635c608591bcd80aed88e5b4a14d1b2ca1da46518ee6: Status 404 returned error can't find the container with id 3bd89a2e73cc40476778635c608591bcd80aed88e5b4a14d1b2ca1da46518ee6 Jul 21 21:48:39.654151 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.654138 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:48:39.662399 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.662355 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" event={"ID":"b3ed946aa6fec2dea4a6cd621aaa10c4","Type":"ContainerStarted","Data":"3bd89a2e73cc40476778635c608591bcd80aed88e5b4a14d1b2ca1da46518ee6"} Jul 21 21:48:39.663364 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.663343 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" event={"ID":"0d007f4d04ced0c10ebffd3e44178333","Type":"ContainerStarted","Data":"95545d65b07310a2226f8d8fff848abd004dc8fc927e7ef9c14d43590b784450"} Jul 21 21:48:39.722493 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.722449 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.782450 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.782358 2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:39.824436 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.824403 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.926712 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:39.926680 2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-224.ec2.internal\" not found" Jul 21 21:48:39.930845 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.930827 2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:39.987386 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.987355 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" Jul 21 21:48:39.998533 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.998505 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:39.999428 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:39.999410 2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" Jul 21 21:48:40.010101 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.010075 2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:40.467161 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.467121 2555 apiserver.go:52] "Watching apiserver" Jul 21 21:48:40.475083 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.475051 2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 21:48:40.475636 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.475595 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9khl7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8","openshift-cluster-node-tuning-operator/tuned-f4lzt","openshift-dns/node-resolver-dxh8g","openshift-image-registry/node-ca-6v4wq","openshift-multus/multus-additional-cni-plugins-2n7nf","openshift-multus/multus-tftsp","openshift-multus/network-metrics-daemon-85x75","openshift-network-diagnostics/network-check-target-8gzkz","kube-system/global-pull-secret-syncer-tfqbs","kube-system/konnectivity-agent-dsqzk","kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal","openshift-network-operator/iptables-alerter-56sc9"] Jul 21 21:48:40.478610 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.478586 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.481217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481193 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.481519 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481500 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 21:48:40.482028 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481820 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.482028 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481867 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmdzp\"" Jul 21 21:48:40.482028 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481918 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 21:48:40.482028 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.481953 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.484418 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.484395 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 21:48:40.484660 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.484636 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r87v9\"" Jul 21 21:48:40.484830 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.484813 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 21:48:40.484982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.484960 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.487314 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.487244 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.487314 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.487262 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.487605 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.487583 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f84mv\"" Jul 21 21:48:40.487745 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.487729 2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:40.488154 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.487920 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.489797 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.489641 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.489797 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.489785 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.490176 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.490143 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4twls\"" Jul 21 21:48:40.490412 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.490395 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.492118 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.492095 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.492379 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.492358 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.492464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.492388 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8x72r\"" Jul 21 21:48:40.492464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.492421 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 21:48:40.492712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.492694 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.494800 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.494777 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.494902 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.494873 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:40.495078 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.495035 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:40.495258 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.495241 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.495258 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.495249 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 21:48:40.495408 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.495363 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blt9n\"" Jul 21 21:48:40.497658 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.497638 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.497772 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.497709 2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:40.500269 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.500248 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.501033 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501010 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-os-release\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501111 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501053 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbv4\" (UniqueName: \"kubernetes.io/projected/4624a09f-69ea-453d-8828-459d0d131dc5-kube-api-access-4bbv4\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501111 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501080 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-kubernetes\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501111 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501105 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-systemd\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501127 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-run\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501152 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-conf-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501177 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501201 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-sys\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501205 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501224 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmqsp\" (UniqueName: \"kubernetes.io/projected/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-kube-api-access-pmqsp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.501281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501269 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782704f8-becd-4a10-b12f-e9c865f7625f-tmp-dir\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.501652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501289 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.501652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501306 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-system-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501333 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501358 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-multus\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.501912 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501887 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-tuned\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.502017 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501948 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-socket-dir-parent\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.502017 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.501989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-multus-certs\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.502132 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502023 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-conf\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.502132 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502051 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-lib-modules\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.502132 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502086 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-cni-binary-copy\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.502132 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502119 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-bin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.502371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502153 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysconfig\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.502371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502186 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f536e43-bf21-4c0f-b058-cbdf14b56af9-host\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.502514 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502393 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 21:48:40.502954 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.502932 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 21:48:40.503272 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.503254 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 21:48:40.504103 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.504080 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.504621 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.504601 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjv9w\"" Jul 21 21:48:40.504911 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.504883 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.504989 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.504962 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:40.505249 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505224 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-cnibin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505333 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505267 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-hostroot\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505333 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505300 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-agent-certs\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.505433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782704f8-becd-4a10-b12f-e9c865f7625f-hosts-file\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.505433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505369 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f536e43-bf21-4c0f-b058-cbdf14b56af9-serviceca\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.505433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505402 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x8j\" (UniqueName: \"kubernetes.io/projected/6f536e43-bf21-4c0f-b058-cbdf14b56af9-kube-api-access-g5x8j\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505431 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-k8s-cni-cncf-io\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-kubelet\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505499 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-multus-daemon-config\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505530 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-konnectivity-ca\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505562 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79kb\" (UniqueName: \"kubernetes.io/projected/782704f8-becd-4a10-b12f-e9c865f7625f-kube-api-access-r79kb\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505589 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-netns\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505609 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505620 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-etc-kubernetes\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.505980 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505651 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-modprobe-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.505980 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505748 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-var-lib-kubelet\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.505980 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505783 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-host\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.505980 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.505810 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-tmp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.507417 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.506907 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 21:48:40.507417 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.507228 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 21:48:40.507586 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.507570 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 21:48:40.507663 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.507573 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d856x\"" Jul 21 21:48:40.507832 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.507766 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.508773 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.508469 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:40.511156 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.511125 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.513739 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.513716 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 21:48:40.513834 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.513752 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9qb2v\"" Jul 21 21:48:40.513977 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.513960 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 21:48:40.523046 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.523025 2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hkk8" Jul 21 21:48:40.531452 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.531428 2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hkk8" Jul 21 21:48:40.589128 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.589103 2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 21:48:40.606777 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.606743 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-var-lib-kubelet\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.606968 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.606784 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-tmp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.606968 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.606888 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-var-lib-kubelet\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.606968 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.606952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-systemd-units\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607191 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.606983 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-log-socket\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607191 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607054 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7bk\" (UniqueName: \"kubernetes.io/projected/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-kube-api-access-tc7bk\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607191 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607093 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-os-release\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607191 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607123 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-systemd\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607191 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607157 2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607205 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-os-release\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607216 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-systemd\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607237 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607259 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmqsp\" (UniqueName: \"kubernetes.io/projected/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-kube-api-access-pmqsp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607283 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-kubelet-config\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607309 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovn-node-metrics-cert\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.607405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607380 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607406 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-multus\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607432 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-tuned\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607456 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5m5q\" (UniqueName: \"kubernetes.io/projected/aba52102-36ce-4326-9452-0e4456e6f912-kube-api-access-z5m5q\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607480 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-systemd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607508 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-multus-certs\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607532 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-conf\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607558 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-lib-modules\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607592 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-kubelet\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607617 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysconfig\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607642 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f536e43-bf21-4c0f-b058-cbdf14b56af9-host\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607692 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782704f8-becd-4a10-b12f-e9c865f7625f-hosts-file\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x8j\" (UniqueName: \"kubernetes.io/projected/6f536e43-bf21-4c0f-b058-cbdf14b56af9-kube-api-access-g5x8j\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607723 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607812 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-cnibin\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607834 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782704f8-becd-4a10-b12f-e9c865f7625f-hosts-file\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607853 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-k8s-cni-cncf-io\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.607864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607861 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-multus\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607816 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysconfig\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607882 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-kubelet\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607895 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-sysctl-conf\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607901 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-multus-certs\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607914 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-konnectivity-ca\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607918 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-k8s-cni-cncf-io\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607921 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607943 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.607948 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-kubelet\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608069 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f536e43-bf21-4c0f-b058-cbdf14b56af9-host\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608195 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-lib-modules\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608207 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-node-log\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608300 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608334 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-bin\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608363 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-sys-fs\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608413 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-netns\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.608629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608443 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-etc-kubernetes\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-modprobe-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608480 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-run-netns\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608497 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-host\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608522 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-slash\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608522 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-etc-kubernetes\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608582 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-host\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608604 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-modprobe-d\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608665 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-registration-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608692 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608738 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbv4\" (UniqueName: \"kubernetes.io/projected/4624a09f-69ea-453d-8828-459d0d131dc5-kube-api-access-4bbv4\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608768 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-kubernetes\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608793 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-run\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608800 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-konnectivity-ca\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608840 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-run\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608887 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-kubernetes\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608956 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-binary-copy\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.609486 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.608996 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609070 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-netd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609101 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-conf-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609129 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-sys\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609146 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-conf-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609157 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782704f8-becd-4a10-b12f-e9c865f7625f-tmp-dir\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609196 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609204 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-sys\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609221 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609246 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-env-overrides\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-socket-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609294 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-device-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609322 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-system-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6zp\" (UniqueName: \"kubernetes.io/projected/6aebda04-dee6-4b4f-8434-721d2c1a5936-kube-api-access-jl6zp\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609374 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqfx\" (UniqueName: \"kubernetes.io/projected/e67ae675-f67c-4234-9ee4-1296aaed5ca5-kube-api-access-vdqfx\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-ovn\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.610312 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609425 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-script-lib\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609453 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnxj\" (UniqueName: \"kubernetes.io/projected/976fb8a5-6141-4dc1-8771-74597b154d16-kube-api-access-xpnxj\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609479 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-socket-dir-parent\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609484 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782704f8-becd-4a10-b12f-e9c865f7625f-tmp-dir\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609502 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-dbus\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609531 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-os-release\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609557 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e67ae675-f67c-4234-9ee4-1296aaed5ca5-iptables-alerter-script\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609582 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e67ae675-f67c-4234-9ee4-1296aaed5ca5-host-slash\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609606 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-netns\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609625 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-system-cni-dir\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609703 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-multus-socket-dir-parent\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609707 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-var-lib-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609794 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609824 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-cni-binary-copy\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609849 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-bin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609874 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-system-cni-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.611214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609902 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609927 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-config\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609954 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-cnibin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.609928 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-host-var-lib-cni-bin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610000 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-hostroot\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610022 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-cnibin\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610040 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-agent-certs\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610065 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4624a09f-69ea-453d-8828-459d0d131dc5-hostroot\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610069 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f536e43-bf21-4c0f-b058-cbdf14b56af9-serviceca\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610100 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610217 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-multus-daemon-config\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610247 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79kb\" (UniqueName: \"kubernetes.io/projected/782704f8-becd-4a10-b12f-e9c865f7625f-kube-api-access-r79kb\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610275 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-etc-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610437 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-cni-binary-copy\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.610571 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f536e43-bf21-4c0f-b058-cbdf14b56af9-serviceca\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.612071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.611502 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4624a09f-69ea-453d-8828-459d0d131dc5-multus-daemon-config\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.612844 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.612541 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-etc-tuned\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.612844 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.612769 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-tmp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.614061 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.614035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7-agent-certs\") pod \"konnectivity-agent-dsqzk\" (UID: \"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7\") " pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.617104 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.617037 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x8j\" (UniqueName: \"kubernetes.io/projected/6f536e43-bf21-4c0f-b058-cbdf14b56af9-kube-api-access-g5x8j\") pod \"node-ca-6v4wq\" (UID: \"6f536e43-bf21-4c0f-b058-cbdf14b56af9\") " pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.618041 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.618015 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmqsp\" (UniqueName: \"kubernetes.io/projected/5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5-kube-api-access-pmqsp\") pod \"tuned-f4lzt\" (UID: \"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5\") " pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.618496 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.618468 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbv4\" (UniqueName: \"kubernetes.io/projected/4624a09f-69ea-453d-8828-459d0d131dc5-kube-api-access-4bbv4\") pod \"multus-tftsp\" (UID: \"4624a09f-69ea-453d-8828-459d0d131dc5\") " pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.619458 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.619399 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79kb\" (UniqueName: \"kubernetes.io/projected/782704f8-becd-4a10-b12f-e9c865f7625f-kube-api-access-r79kb\") pod \"node-resolver-dxh8g\" (UID: \"782704f8-becd-4a10-b12f-e9c865f7625f\") " pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.653596 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.653570 2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:40.710694 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-system-cni-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.710694 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710689 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710720 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-config\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710752 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-system-cni-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710780 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-etc-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710829 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-systemd-units\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710878 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-systemd-units\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710888 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.710946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710933 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-etc-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710979 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-log-socket\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.710974 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-log-socket\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711023 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7bk\" (UniqueName: \"kubernetes.io/projected/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-kube-api-access-tc7bk\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-kubelet-config\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711080 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovn-node-metrics-cert\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711104 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711133 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5m5q\" (UniqueName: \"kubernetes.io/projected/aba52102-36ce-4326-9452-0e4456e6f912-kube-api-access-z5m5q\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711178 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-kubelet-config\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711186 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.711310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711292 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-systemd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711326 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-kubelet\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711370 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-cnibin\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711392 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-systemd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711423 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711435 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-kubelet\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711399 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711464 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-config\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.711484 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711485 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-node-log\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711472 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-cnibin\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711521 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711530 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-node-log\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711552 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-bin\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711583 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-ovn-kubernetes\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711595 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-bin\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.711594 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:41.211568908 +0000 UTC m=+3.062328737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:40.711868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711649 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-sys-fs\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711683 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-slash\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711707 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-registration-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711731 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711744 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-slash\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711761 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-binary-copy\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711761 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-sys-fs\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711799 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711802 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-registration-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711841 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711870 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711875 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-netd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711993 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-env-overrides\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712025 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-socket-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712052 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-device-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.712805 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.712082 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712137 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.712154 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:41.212134838 +0000 UTC m=+3.062894668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6zp\" (UniqueName: \"kubernetes.io/projected/6aebda04-dee6-4b4f-8434-721d2c1a5936-kube-api-access-jl6zp\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712197 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqfx\" (UniqueName: \"kubernetes.io/projected/e67ae675-f67c-4234-9ee4-1296aaed5ca5-kube-api-access-vdqfx\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712231 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-ovn\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712258 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-script-lib\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712274 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-cni-binary-copy\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712319 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6aebda04-dee6-4b4f-8434-721d2c1a5936-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712345 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-device-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.711993 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-cni-netd\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712394 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/976fb8a5-6141-4dc1-8771-74597b154d16-socket-dir\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712409 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-ovn\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712436 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnxj\" (UniqueName: \"kubernetes.io/projected/976fb8a5-6141-4dc1-8771-74597b154d16-kube-api-access-xpnxj\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712472 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-dbus\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712500 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-os-release\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.713568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712518 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-env-overrides\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712528 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e67ae675-f67c-4234-9ee4-1296aaed5ca5-iptables-alerter-script\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712557 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0298deea-f91e-4d59-9170-a1914c0e4491-dbus\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712568 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e67ae675-f67c-4234-9ee4-1296aaed5ca5-host-slash\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-netns\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712603 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6aebda04-dee6-4b4f-8434-721d2c1a5936-os-release\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712604 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e67ae675-f67c-4234-9ee4-1296aaed5ca5-host-slash\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712619 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-var-lib-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712647 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712656 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-host-run-netns\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712714 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-var-lib-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712726 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-run-openvswitch\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.712817 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovnkube-script-lib\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.713101 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e67ae675-f67c-4234-9ee4-1296aaed5ca5-iptables-alerter-script\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.714289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.713907 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-ovn-node-metrics-cert\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.719662 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.719549 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:40.719662 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.719583 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:40.719662 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.719598 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:40.719889 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:40.719683 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:41.219661894 +0000 UTC m=+3.070421721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:40.722171 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.722139 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5m5q\" (UniqueName: \"kubernetes.io/projected/aba52102-36ce-4326-9452-0e4456e6f912-kube-api-access-z5m5q\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:40.722631 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.722602 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6zp\" (UniqueName: \"kubernetes.io/projected/6aebda04-dee6-4b4f-8434-721d2c1a5936-kube-api-access-jl6zp\") pod \"multus-additional-cni-plugins-2n7nf\" (UID: \"6aebda04-dee6-4b4f-8434-721d2c1a5936\") " pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:40.722732 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.722685 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7bk\" (UniqueName: \"kubernetes.io/projected/90c9984c-70a9-4ad4-9f40-f7c412a1cc57-kube-api-access-tc7bk\") pod \"ovnkube-node-9khl7\" (UID: \"90c9984c-70a9-4ad4-9f40-f7c412a1cc57\") " pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.723299 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.723277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnxj\" (UniqueName: \"kubernetes.io/projected/976fb8a5-6141-4dc1-8771-74597b154d16-kube-api-access-xpnxj\") pod \"aws-ebs-csi-driver-node-zxnn8\" (UID: \"976fb8a5-6141-4dc1-8771-74597b154d16\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.723384 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.723277 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqfx\" (UniqueName: \"kubernetes.io/projected/e67ae675-f67c-4234-9ee4-1296aaed5ca5-kube-api-access-vdqfx\") pod \"iptables-alerter-56sc9\" (UID: \"e67ae675-f67c-4234-9ee4-1296aaed5ca5\") " pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.792354 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.792310 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tftsp" Jul 21 21:48:40.809262 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.809237 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:48:40.818329 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.818308 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" Jul 21 21:48:40.825218 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.825197 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxh8g" Jul 21 21:48:40.834086 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.834062 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6v4wq" Jul 21 21:48:40.843149 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.843129 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-56sc9" Jul 21 21:48:40.850980 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.850959 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:48:40.859790 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.859765 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" Jul 21 21:48:40.865761 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:40.865737 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" Jul 21 21:48:41.216322 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.216199 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:41.216322 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.216296 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:41.216562 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.216355 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:41.216562 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.216404 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:41.216562 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.216446 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:42.216423615 +0000 UTC m=+4.067183432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:41.216562 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.216471 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:42.216453475 +0000 UTC m=+4.067213291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:41.317238 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.317183 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:41.317439 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.317325 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:41.317439 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.317349 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:41.317439 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.317364 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:41.317439 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.317437 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:42.317419596 +0000 UTC m=+4.168179406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:41.523092 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.523006 2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:41.535433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.535390 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:40 +0000 UTC" deadline="2027-12-20 01:05:48.35226511 +0000 UTC" Jul 21 21:48:41.535433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.535429 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12387h17m6.816840926s" Jul 21 21:48:41.658469 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:41.658438 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:41.658669 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:41.658573 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:41.722981 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.722853 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f536e43_bf21_4c0f_b058_cbdf14b56af9.slice/crio-a86ac4696cba424ebc52b6aae2d56187d9328c014e726bf1f6d07ca915be1cc6 WatchSource:0}: Error finding container a86ac4696cba424ebc52b6aae2d56187d9328c014e726bf1f6d07ca915be1cc6: Status 404 returned error can't find the container with id a86ac4696cba424ebc52b6aae2d56187d9328c014e726bf1f6d07ca915be1cc6 Jul 21 21:48:41.727104 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.727073 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6171ab4d_bb3d_4bf3_95f8_2722bc2e83c7.slice/crio-9bb8f9383d227d82743b45bb02d4c3e9e60a76971fb50acfab1b110a9ae887b0 WatchSource:0}: Error finding container 9bb8f9383d227d82743b45bb02d4c3e9e60a76971fb50acfab1b110a9ae887b0: Status 404 returned error can't find the container with id 9bb8f9383d227d82743b45bb02d4c3e9e60a76971fb50acfab1b110a9ae887b0 Jul 21 21:48:41.728649 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.728574 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aebda04_dee6_4b4f_8434_721d2c1a5936.slice/crio-3784116cd7c33fdf8c23b297974c7128559e86be0f8e05bb4c839e338ee4a258 WatchSource:0}: Error finding container 3784116cd7c33fdf8c23b297974c7128559e86be0f8e05bb4c839e338ee4a258: Status 404 returned error can't find the container with id 3784116cd7c33fdf8c23b297974c7128559e86be0f8e05bb4c839e338ee4a258 Jul 21 21:48:41.729820 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.729797 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c9984c_70a9_4ad4_9f40_f7c412a1cc57.slice/crio-780971260e38c78cef82888a62cabf7665f8895571c3c6cede1e6cd03f835484 WatchSource:0}: Error finding container 780971260e38c78cef82888a62cabf7665f8895571c3c6cede1e6cd03f835484: Status 404 returned error can't find the container with id 780971260e38c78cef82888a62cabf7665f8895571c3c6cede1e6cd03f835484 Jul 21 21:48:41.731353 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.731327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4624a09f_69ea_453d_8828_459d0d131dc5.slice/crio-0b401c30bc9bf48329d7787d5a6ab4b0fa5236a9c6a4cd5121ac8c1a36129092 WatchSource:0}: Error finding container 0b401c30bc9bf48329d7787d5a6ab4b0fa5236a9c6a4cd5121ac8c1a36129092: Status 404 returned error can't find the container with id 0b401c30bc9bf48329d7787d5a6ab4b0fa5236a9c6a4cd5121ac8c1a36129092 Jul 21 21:48:41.732762 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.732733 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782704f8_becd_4a10_b12f_e9c865f7625f.slice/crio-e516cb5a68d3441f7d1de0d05c8567ad2e6195882c10dd1fe98f362d0b9be268 WatchSource:0}: Error finding container e516cb5a68d3441f7d1de0d05c8567ad2e6195882c10dd1fe98f362d0b9be268: Status 404 returned error can't find the container with id e516cb5a68d3441f7d1de0d05c8567ad2e6195882c10dd1fe98f362d0b9be268 Jul 21 21:48:41.732968 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.732949 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67ae675_f67c_4234_9ee4_1296aaed5ca5.slice/crio-1c72b708e5be02f472dc1160cb89ee5bfbc4dc9df70042da2d41d851866b32da WatchSource:0}: Error finding container 1c72b708e5be02f472dc1160cb89ee5bfbc4dc9df70042da2d41d851866b32da: Status 404 returned error can't find the container with id 1c72b708e5be02f472dc1160cb89ee5bfbc4dc9df70042da2d41d851866b32da Jul 21 21:48:41.735019 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.734892 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976fb8a5_6141_4dc1_8771_74597b154d16.slice/crio-181e612f7f1293f2948d377a403eba05096970a3d2a4a77800e2b09c9dd3aa97 WatchSource:0}: Error finding container 181e612f7f1293f2948d377a403eba05096970a3d2a4a77800e2b09c9dd3aa97: Status 404 returned error can't find the container with id 181e612f7f1293f2948d377a403eba05096970a3d2a4a77800e2b09c9dd3aa97 Jul 21 21:48:41.757929 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:48:41.757901 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5e17da_a5bd_4ef9_8b32_c4fbf66b99e5.slice/crio-91830dc9bc873ee10da3e1a75f625ea69a05c62bdae08503e659fbca40597f4a WatchSource:0}: Error finding container 91830dc9bc873ee10da3e1a75f625ea69a05c62bdae08503e659fbca40597f4a: Status 404 returned error can't find the container with id 91830dc9bc873ee10da3e1a75f625ea69a05c62bdae08503e659fbca40597f4a Jul 21 21:48:42.224271 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.223881 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:42.224271 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.224252 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:42.224479 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.224402 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:42.224536 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.224466 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:44.22444673 +0000 UTC m=+6.075206542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:42.224732 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.224705 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:42.224874 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.224773 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:44.224755232 +0000 UTC m=+6.075515066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:42.325258 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.325173 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:42.325430 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.325380 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:42.325430 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.325406 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:42.325430 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.325423 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:42.325636 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.325484 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:44.32546546 +0000 UTC m=+6.176225281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:42.536044 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.535945 2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:40 +0000 UTC" deadline="2028-04-26 17:10:17.723823636 +0000 UTC" Jul 21 21:48:42.536044 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.536004 2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15475h21m35.18782355s" Jul 21 21:48:42.659757 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.659726 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:42.659917 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.659726 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:42.659917 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.659863 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:42.659917 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:42.659901 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:42.683477 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.683370 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"780971260e38c78cef82888a62cabf7665f8895571c3c6cede1e6cd03f835484"} Jul 21 21:48:42.688858 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.688772 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerStarted","Data":"3784116cd7c33fdf8c23b297974c7128559e86be0f8e05bb4c839e338ee4a258"} Jul 21 21:48:42.693106 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.693031 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsqzk" event={"ID":"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7","Type":"ContainerStarted","Data":"9bb8f9383d227d82743b45bb02d4c3e9e60a76971fb50acfab1b110a9ae887b0"} Jul 21 21:48:42.695652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.695563 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6v4wq" event={"ID":"6f536e43-bf21-4c0f-b058-cbdf14b56af9","Type":"ContainerStarted","Data":"a86ac4696cba424ebc52b6aae2d56187d9328c014e726bf1f6d07ca915be1cc6"} Jul 21 21:48:42.717718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.717680 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" event={"ID":"b3ed946aa6fec2dea4a6cd621aaa10c4","Type":"ContainerStarted","Data":"59d255677d3b23aed24558ef7b4d03aba83933829e856af9029bcf4e42721f61"} Jul 21 21:48:42.729356 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.729320 2555 generic.go:356] "Generic (PLEG): container finished" podID="0d007f4d04ced0c10ebffd3e44178333" containerID="ad6ad047550ebc8b9c673bacca68f55666e28f8b1676a4da20e592437936d868" exitCode=0 Jul 21 21:48:42.729527 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.729414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" event={"ID":"0d007f4d04ced0c10ebffd3e44178333","Type":"ContainerDied","Data":"ad6ad047550ebc8b9c673bacca68f55666e28f8b1676a4da20e592437936d868"} Jul 21 21:48:42.738367 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.738329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" event={"ID":"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5","Type":"ContainerStarted","Data":"91830dc9bc873ee10da3e1a75f625ea69a05c62bdae08503e659fbca40597f4a"} Jul 21 21:48:42.748977 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.748919 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-224.ec2.internal" podStartSLOduration=2.748893923 podStartE2EDuration="2.748893923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:42.733899336 +0000 UTC m=+4.584659167" watchObservedRunningTime="2026-07-21 21:48:42.748893923 +0000 UTC m=+4.599653767" Jul 21 21:48:42.750749 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.750716 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" event={"ID":"976fb8a5-6141-4dc1-8771-74597b154d16","Type":"ContainerStarted","Data":"181e612f7f1293f2948d377a403eba05096970a3d2a4a77800e2b09c9dd3aa97"} Jul 21 21:48:42.759706 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.759667 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56sc9" event={"ID":"e67ae675-f67c-4234-9ee4-1296aaed5ca5","Type":"ContainerStarted","Data":"1c72b708e5be02f472dc1160cb89ee5bfbc4dc9df70042da2d41d851866b32da"} Jul 21 21:48:42.772300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.772263 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxh8g" event={"ID":"782704f8-becd-4a10-b12f-e9c865f7625f","Type":"ContainerStarted","Data":"e516cb5a68d3441f7d1de0d05c8567ad2e6195882c10dd1fe98f362d0b9be268"} Jul 21 21:48:42.773628 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:42.773588 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tftsp" event={"ID":"4624a09f-69ea-453d-8828-459d0d131dc5","Type":"ContainerStarted","Data":"0b401c30bc9bf48329d7787d5a6ab4b0fa5236a9c6a4cd5121ac8c1a36129092"} Jul 21 21:48:43.658575 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:43.658543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:43.659044 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:43.658711 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:43.782261 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:43.782219 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" event={"ID":"0d007f4d04ced0c10ebffd3e44178333","Type":"ContainerStarted","Data":"0e736430570b6f5f4d81b85ca3d2b199f1944280b94857b34c0aa1983dbb3db3"} Jul 21 21:48:43.797708 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:43.797651 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-224.ec2.internal" podStartSLOduration=3.7976288719999998 podStartE2EDuration="3.797628872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.79731805 +0000 UTC m=+5.648077882" watchObservedRunningTime="2026-07-21 21:48:43.797628872 +0000 UTC m=+5.648388703" Jul 21 21:48:44.243160 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:44.243123 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:44.243351 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:44.243203 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:44.243415 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.243387 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:44.243473 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.243449 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.243432087 +0000 UTC m=+10.094191910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:44.243992 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.243950 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:44.244078 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.244003 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.243988021 +0000 UTC m=+10.094747834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:44.344184 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:44.344143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:44.344383 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.344363 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:44.344461 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.344450 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:44.344509 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.344470 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:44.344565 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.344532 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:48.344511588 +0000 UTC m=+10.195271399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:44.659806 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:44.659696 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:44.660237 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.659828 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:44.660655 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:44.660361 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:44.660655 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:44.660560 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:45.658787 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:45.658726 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:45.658975 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:45.658871 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:46.658805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:46.658775 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:46.658805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:46.658789 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:46.659294 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:46.658910 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:46.659294 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:46.659062 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:47.658971 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:47.658919 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:47.659421 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:47.659065 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:48.279968 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:48.279916 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:48.280148 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:48.279990 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:48.280227 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.280153 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:48.280227 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.280219 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:56.280199189 +0000 UTC m=+18.130959001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:48.280533 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.280443 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:48.280533 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.280497 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:56.280482569 +0000 UTC m=+18.131242382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:48.381240 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:48.381140 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:48.381418 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.381333 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:48.381418 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.381360 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:48.381418 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.381372 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:48.381585 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.381435 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:56.38141642 +0000 UTC m=+18.232176244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:48.660453 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:48.659965 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:48.660453 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.660077 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:48.660453 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:48.660190 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:48.660453 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:48.660274 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:49.658750 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:49.658716 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:49.658935 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:49.658847 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:50.659250 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:50.659187 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:50.659739 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:50.659322 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:50.659739 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:50.659378 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:50.659739 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:50.659492 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:51.658718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:51.658677 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:51.658908 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:51.658836 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:52.659338 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:52.659308 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:52.659792 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:52.659323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:52.659792 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:52.659453 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:52.659792 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:52.659504 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:53.658900 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:53.658866 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:53.659105 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:53.658990 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:54.659114 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:54.659075 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:54.659581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:54.659129 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:54.659581 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:54.659214 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:54.659581 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:54.659339 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:55.659058 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:55.659015 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:55.659349 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:55.659180 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:56.345190 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:56.345158 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:56.345342 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:56.345211 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:56.345342 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.345305 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:56.345342 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.345306 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:56.345442 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.345357 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.345342739 +0000 UTC m=+34.196102547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:56.345442 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.345374 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.345365136 +0000 UTC m=+34.196124949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:56.446021 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:56.445987 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:56.446182 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.446144 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:56.446182 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.446163 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:56.446182 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.446173 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:56.446300 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.446220 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.446207765 +0000 UTC m=+34.296967574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:56.658794 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:56.658702 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:56.658958 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.658849 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:56.658958 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:56.658907 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:56.659074 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:56.659023 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:57.659058 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:57.659020 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:57.659560 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:57.659160 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:48:58.659744 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:58.659658 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:48:58.660218 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:58.659775 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:48:58.660218 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:58.659828 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:48:58.660218 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:58.659933 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:48:59.659382 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:48:59.659344 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:48:59.659551 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:48:59.659494 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:00.659395 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.658960 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:00.660095 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.658961 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:00.660095 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:00.659522 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:00.660095 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:00.659627 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:00.822455 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.822424 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" event={"ID":"5e5e17da-a5bd-4ef9-8b32-c4fbf66b99e5","Type":"ContainerStarted","Data":"4f7e39225dee1b15f2e1830cd96bcd67a70d97dea25b94830dda16b71ad32df5"} Jul 21 21:49:00.823764 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.823739 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" event={"ID":"976fb8a5-6141-4dc1-8771-74597b154d16","Type":"ContainerStarted","Data":"8fbed1c4a174a30f8b8d8d12fd149b31e985c1f0634fe5d9f9e8905742dcb262"} Jul 21 21:49:00.824986 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.824965 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxh8g" event={"ID":"782704f8-becd-4a10-b12f-e9c865f7625f","Type":"ContainerStarted","Data":"d9a2c9d085b22e557385bdd8a0d266a4d5f3b9d58f5d4bf99d3144860fb47310"} Jul 21 21:49:00.826327 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.826307 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tftsp" event={"ID":"4624a09f-69ea-453d-8828-459d0d131dc5","Type":"ContainerStarted","Data":"2f74d34f89499cc0939abdaa324b05b68479b3484989b6d8640d28cde659309b"} Jul 21 21:49:00.828421 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.828398 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"86b3266bd6249e96c56417af853e9048fb69253f4736e5d841c954b9ef15bdcb"} Jul 21 21:49:00.828523 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.828452 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"a7ad2dc2dc90c5a5209988b375a293151c414deb1ed221e8b1105ee195e5de66"} Jul 21 21:49:00.828523 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.828481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"192c62d55e3506440b6bec1b3faf929cb0572b44f6e7d7a13bb6ec975140f2cf"} Jul 21 21:49:00.829728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.829708 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="6d8c10e63c69f6262e1510ea6ece1abc2de94e3e6ad74904821f6a285c9b59c8" exitCode=0 Jul 21 21:49:00.829787 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.829770 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"6d8c10e63c69f6262e1510ea6ece1abc2de94e3e6ad74904821f6a285c9b59c8"} Jul 21 21:49:00.831165 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.831146 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsqzk" event={"ID":"6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7","Type":"ContainerStarted","Data":"37e5ee6f1ed5fda6c871c84325b4d310027aad8fe610ee91186076670a8f68d6"} Jul 21 21:49:00.832426 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.832409 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6v4wq" event={"ID":"6f536e43-bf21-4c0f-b058-cbdf14b56af9","Type":"ContainerStarted","Data":"59256d8cc4625d5f5b671bfea36b4be862df19855d272a7985d7dc16089bc0a0"} Jul 21 21:49:00.840410 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.840376 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f4lzt" podStartSLOduration=4.262047592 podStartE2EDuration="22.840365065s" totalImagesPullingTime="18.578317473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.840212419 +0000 UTC m=+22.690972247" watchObservedRunningTime="2026-07-21 21:49:00.840365065 +0000 UTC m=+22.691124896" Jul 21 21:49:00.854796 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.854692 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dsqzk" podStartSLOduration=12.464101958 podStartE2EDuration="22.854678406s" totalImagesPullingTime="10.390576448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.854459461 +0000 UTC m=+22.705219293" watchObservedRunningTime="2026-07-21 21:49:00.854678406 +0000 UTC m=+22.705438236" Jul 21 21:49:00.894893 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.894849 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tftsp" podStartSLOduration=4.231656681 podStartE2EDuration="22.894832922s" totalImagesPullingTime="18.663176241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.894483826 +0000 UTC m=+22.745243656" watchObservedRunningTime="2026-07-21 21:49:00.894832922 +0000 UTC m=+22.745592753" Jul 21 21:49:00.913434 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.913386 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dxh8g" podStartSLOduration=4.334636175 podStartE2EDuration="22.913367308s" totalImagesPullingTime="18.578731133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.912928776 +0000 UTC m=+22.763688607" watchObservedRunningTime="2026-07-21 21:49:00.913367308 +0000 UTC m=+22.764127141" Jul 21 21:49:00.926697 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:00.926656 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6v4wq" podStartSLOduration=4.315211865 podStartE2EDuration="22.926642465s" totalImagesPullingTime="18.6114306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.926517594 +0000 UTC m=+22.777277424" watchObservedRunningTime="2026-07-21 21:49:00.926642465 +0000 UTC m=+22.777402295" Jul 21 21:49:01.056134 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.056097 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:01.128725 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.128639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:01.658874 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.658835 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:01.659049 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:01.658965 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:01.823221 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.823194 2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 21:49:01.839624 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.839576 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"592f071590d562b082fef7ebd0741ac5049a4100915043dbeda43913665b7e60"} Jul 21 21:49:01.839746 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.839665 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"54ba67d96f942c4a6d7f96b808eaab8eb1ca7e45ea7be5de1809c9f5d70ea707"} Jul 21 21:49:01.839746 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.839709 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"a60eae21cbac59dac67f94f2d0b3a7a2e3b6a5517450d579045c4963b9737cdf"} Jul 21 21:49:01.841536 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.841512 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" event={"ID":"976fb8a5-6141-4dc1-8771-74597b154d16","Type":"ContainerStarted","Data":"3d29c915f76cf4e260ae593f2e76bf001e9a54d78ce77894d2823a60af2fb5d8"} Jul 21 21:49:01.843277 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.843226 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-56sc9" event={"ID":"e67ae675-f67c-4234-9ee4-1296aaed5ca5","Type":"ContainerStarted","Data":"826b0056ee1a6198ed0be08a085d646c2e7aabe3501684e5d660abe7334a5b5b"} Jul 21 21:49:01.860926 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.860874 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-56sc9" podStartSLOduration=5.282140636 podStartE2EDuration="23.860856079s" totalImagesPullingTime="18.578715443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:01.860324832 +0000 UTC m=+23.711084665" watchObservedRunningTime="2026-07-21 21:49:01.860856079 +0000 UTC m=+23.711615909" Jul 21 21:49:01.984143 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:01.984108 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:02.037567 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.037533 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:02.425281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.425197 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:49:02.425988 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.425962 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:49:02.570209 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.569974 2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T21:49:01.823215334Z","UUID":"1e42e309-06e2-49ae-9f18-b440030a2fbb","Handler":null,"Name":"","Endpoint":""} Jul 21 21:49:02.571904 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.571874 2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 21:49:02.572033 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.571917 2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 21:49:02.659384 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.659347 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:02.659577 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:02.659494 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:02.659740 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.659707 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:02.659867 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:02.659825 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:02.719123 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.719047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:02.777878 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.777834 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:02.845622 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.845385 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:49:02.846063 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.845968 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dsqzk" Jul 21 21:49:02.916188 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:02.916159 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:03.019008 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.018910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:03.075534 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.075500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:03.659089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.659007 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:03.659310 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:03.659119 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:03.850129 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.850093 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" event={"ID":"976fb8a5-6141-4dc1-8771-74597b154d16","Type":"ContainerStarted","Data":"3595c7991d96d7214e4dd79882fbd2ed7d1e8639d3e88d7d64d6690d76e57b00"} Jul 21 21:49:03.854046 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.854015 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"b34d6f1e845ad5b9a6f4c8a45946fd33060953de8a386e0a32da14318f5d8e94"} Jul 21 21:49:03.870372 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:03.870323 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxnn8" podStartSLOduration=4.783208779 podStartE2EDuration="25.87030204s" totalImagesPullingTime="21.087093261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:03.869831062 +0000 UTC m=+25.720590894" watchObservedRunningTime="2026-07-21 21:49:03.87030204 +0000 UTC m=+25.721061872" Jul 21 21:49:04.659367 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:04.659131 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:04.659560 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:04.659200 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:04.659560 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:04.659482 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:04.659724 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:04.659586 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:04.817386 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:04.817345 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:04.918238 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:04.918150 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:05.659530 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.659304 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:05.659690 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:05.659567 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:05.862360 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.862322 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" event={"ID":"90c9984c-70a9-4ad4-9f40-f7c412a1cc57","Type":"ContainerStarted","Data":"f7e7c3e73ce0247a7a3145672978d49c2907ef742e3001729a21ac5f964e3498"} Jul 21 21:49:05.862581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.862561 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:05.864289 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.864259 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="3cd3b5196d04354ac9e238849a329bf18ad618c409366112fab8c2de4c13316f" exitCode=0 Jul 21 21:49:05.864395 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.864318 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"3cd3b5196d04354ac9e238849a329bf18ad618c409366112fab8c2de4c13316f"} Jul 21 21:49:05.878729 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.878703 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:05.905888 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:05.905836 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" podStartSLOduration=9.232432877 podStartE2EDuration="27.905819967s" totalImagesPullingTime="18.67338709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:05.905204926 +0000 UTC m=+27.755964758" watchObservedRunningTime="2026-07-21 21:49:05.905819967 +0000 UTC m=+27.756579798" Jul 21 21:49:06.116647 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.116532 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:06.171875 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.171834 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:06.273524 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.273493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:06.659666 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.659486 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:06.659666 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:06.659640 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:06.660081 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.659486 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:06.660081 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:06.660040 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:06.866164 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.866134 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 21:49:06.866555 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.866536 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:06.881554 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:06.881526 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:07.144472 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.144434 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:07.244484 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.244454 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:07.560228 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.560196 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tfqbs"] Jul 21 21:49:07.560396 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.560354 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:07.560458 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:07.560438 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:07.561853 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.561828 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85x75"] Jul 21 21:49:07.562110 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.562090 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:07.562220 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:07.562201 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:07.562373 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.562359 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gzkz"] Jul 21 21:49:07.562429 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.562419 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:07.562489 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:07.562474 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:07.870387 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.870359 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="dbcbe5d5d225f5c4c1393b58f71c8cfd00d8ff4fc9c2cda3a08518b48fea5228" exitCode=0 Jul 21 21:49:07.870620 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.870441 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"dbcbe5d5d225f5c4c1393b58f71c8cfd00d8ff4fc9c2cda3a08518b48fea5228"} Jul 21 21:49:07.870620 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:07.870504 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 21:49:08.872878 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:08.872718 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 21:49:09.091513 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.091458 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:09.659156 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.659096 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:09.659156 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.659131 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:09.659320 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.659178 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:09.659320 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:09.659190 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:09.659320 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:09.659269 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:09.659427 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:09.659333 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:09.714242 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.714203 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:09.877258 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.877228 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="1b125ebf74cfdd41b3fcbcbad2612f71697cbc534759711cf01da7ddcf044ea0" exitCode=0 Jul 21 21:49:09.877896 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:09.877269 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"1b125ebf74cfdd41b3fcbcbad2612f71697cbc534759711cf01da7ddcf044ea0"} Jul 21 21:49:10.132624 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.132520 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:10.132795 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.132779 2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 21:49:10.148589 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.148547 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" podUID="90c9984c-70a9-4ad4-9f40-f7c412a1cc57" containerName="ovnkube-controller" probeResult="failure" output="" Jul 21 21:49:10.157006 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.156975 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" podUID="90c9984c-70a9-4ad4-9f40-f7c412a1cc57" containerName="ovnkube-controller" probeResult="failure" output="" Jul 21 21:49:10.294174 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.294138 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:10.352799 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:10.352754 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:11.252189 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:11.252161 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxh8g_782704f8-becd-4a10-b12f-e9c865f7625f/dns-node-resolver/0.log" Jul 21 21:49:11.659096 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:11.659015 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:11.659262 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:11.659129 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:11.659262 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:11.659143 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gzkz" podUID="e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0" Jul 21 21:49:11.659262 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:11.659160 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:11.659262 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:11.659254 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tfqbs" podUID="0298deea-f91e-4d59-9170-a1914c0e4491" Jul 21 21:49:11.659476 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:11.659379 2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-85x75" podUID="aba52102-36ce-4326-9452-0e4456e6f912" Jul 21 21:49:12.243695 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:12.243658 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:12.252538 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:12.252513 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6v4wq_6f536e43-bf21-4c0f-b058-cbdf14b56af9/node-ca/0.log" Jul 21 21:49:12.377764 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:12.374937 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:12.377764 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:12.375026 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:12.377998 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.377834 2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:12.377998 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.377974 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret podName:0298deea-f91e-4d59-9170-a1914c0e4491 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:44.377943186 +0000 UTC m=+66.228703013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret") pod "global-pull-secret-syncer-tfqbs" (UID: "0298deea-f91e-4d59-9170-a1914c0e4491") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:49:12.378102 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.377998 2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:12.378399 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.378379 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs podName:aba52102-36ce-4326-9452-0e4456e6f912 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:44.378354571 +0000 UTC m=+66.229114382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs") pod "network-metrics-daemon-85x75" (UID: "aba52102-36ce-4326-9452-0e4456e6f912") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:49:12.476136 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:12.476096 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:12.476315 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.476294 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:49:12.476381 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.476319 2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:49:12.476381 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.476329 2555 projected.go:196] Error preparing data for projected volume kube-api-access-xhlsc for pod openshift-network-diagnostics/network-check-target-8gzkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:12.476485 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:12.476385 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc podName:e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:44.476371435 +0000 UTC m=+66.327131244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xhlsc" (UniqueName: "kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc") pod "network-check-target-8gzkz" (UID: "e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:49:13.000155 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.000118 2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-224.ec2.internal" event="NodeReady" Jul 21 21:49:13.000320 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.000291 2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 21:49:13.039440 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.039409 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:49:13.074869 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.074768 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w5vvr"] Jul 21 21:49:13.075073 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.074896 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.077756 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.077729 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 21:49:13.077887 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.077784 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9js2h\"" Jul 21 21:49:13.077887 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.077830 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 21:49:13.078041 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.078024 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 21:49:13.083459 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.083436 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 21:49:13.088711 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.088689 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d4cg9"] Jul 21 21:49:13.089075 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.088956 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.091529 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.091508 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 21:49:13.091633 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.091509 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp5jq\"" Jul 21 21:49:13.091633 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.091518 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 21:49:13.108695 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.108672 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:49:13.108695 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.108704 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4cg9"] Jul 21 21:49:13.108940 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.108727 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w5vvr"] Jul 21 21:49:13.108940 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.108814 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w8dt8"] Jul 21 21:49:13.108940 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.108858 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.111963 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.111940 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 21:49:13.112245 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.112220 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 21:49:13.112413 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.112392 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7wwbw\"" Jul 21 21:49:13.112524 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.112472 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 21:49:13.128891 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.128866 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8dt8"] Jul 21 21:49:13.129033 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.129000 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.131940 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.131916 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 21:49:13.132065 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.131970 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tqm4\"" Jul 21 21:49:13.132436 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.132280 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 21:49:13.132436 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.132302 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 21:49:13.132570 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.132550 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 21:49:13.181226 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181194 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181401 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181239 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181401 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181266 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181401 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181321 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181439 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181467 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgt99\" (UniqueName: \"kubernetes.io/projected/18704d0c-cabe-4eaf-aa71-73584e07fcc6-kube-api-access-rgt99\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181492 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181508 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rddr\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181526 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18704d0c-cabe-4eaf-aa71-73584e07fcc6-tmp-dir\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.181793 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181583 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.181793 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.181628 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18704d0c-cabe-4eaf-aa71-73584e07fcc6-config-volume\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.187905 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.187863 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:13.283087 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283051 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283128 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283153 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283184 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b938b803-51a6-42c1-b29e-42c7aa84d2eb-data-volume\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283210 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283237 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283261 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgt99\" (UniqueName: \"kubernetes.io/projected/18704d0c-cabe-4eaf-aa71-73584e07fcc6-kube-api-access-rgt99\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283285 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-cert\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283308 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283337 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283362 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rddr\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283384 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18704d0c-cabe-4eaf-aa71-73584e07fcc6-tmp-dir\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283419 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283447 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmnqx\" (UniqueName: \"kubernetes.io/projected/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-api-access-tmnqx\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.283469 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.283492 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85f4b4d747-m9xlq: secret "image-registry-tls" not found Jul 21 21:49:13.283568 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.283531 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.283549 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls podName:7484c312-3831-47c9-9628-8632592ea36c nodeName:}" failed. No retries permitted until 2026-07-21 21:49:13.783530131 +0000 UTC m=+35.634289959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls") pod "image-registry-85f4b4d747-m9xlq" (UID: "7484c312-3831-47c9-9628-8632592ea36c") : secret "image-registry-tls" not found Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.283588 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls podName:18704d0c-cabe-4eaf-aa71-73584e07fcc6 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:13.783570074 +0000 UTC m=+35.634329902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls") pod "dns-default-w5vvr" (UID: "18704d0c-cabe-4eaf-aa71-73584e07fcc6") : secret "dns-default-metrics-tls" not found Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283477 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283785 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18704d0c-cabe-4eaf-aa71-73584e07fcc6-config-volume\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283813 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18704d0c-cabe-4eaf-aa71-73584e07fcc6-tmp-dir\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283817 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgq4\" (UniqueName: \"kubernetes.io/projected/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-kube-api-access-9sgq4\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283871 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b938b803-51a6-42c1-b29e-42c7aa84d2eb-crio-socket\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.283870 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.284365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.284248 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.284728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.284424 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18704d0c-cabe-4eaf-aa71-73584e07fcc6-config-volume\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.284887 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.284867 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.287952 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.287930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.288051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.287930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.293275 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.293238 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgt99\" (UniqueName: \"kubernetes.io/projected/18704d0c-cabe-4eaf-aa71-73584e07fcc6-kube-api-access-rgt99\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.293684 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.293645 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.293893 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.293855 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rddr\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.384983 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.384898 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b938b803-51a6-42c1-b29e-42c7aa84d2eb-data-volume\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.384983 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.384949 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-cert\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.384983 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.384975 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385010 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385037 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmnqx\" (UniqueName: \"kubernetes.io/projected/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-api-access-tmnqx\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgq4\" (UniqueName: \"kubernetes.io/projected/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-kube-api-access-9sgq4\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385087 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b938b803-51a6-42c1-b29e-42c7aa84d2eb-crio-socket\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.385152 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:13.385249 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.385239 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls podName:b938b803-51a6-42c1-b29e-42c7aa84d2eb nodeName:}" failed. No retries permitted until 2026-07-21 21:49:13.885216284 +0000 UTC m=+35.735976112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8dt8" (UID: "b938b803-51a6-42c1-b29e-42c7aa84d2eb") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:13.385533 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385317 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b938b803-51a6-42c1-b29e-42c7aa84d2eb-data-volume\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385533 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385376 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b938b803-51a6-42c1-b29e-42c7aa84d2eb-crio-socket\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.385638 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.385577 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.387970 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.387945 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-cert\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.394700 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.394679 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmnqx\" (UniqueName: \"kubernetes.io/projected/b938b803-51a6-42c1-b29e-42c7aa84d2eb-kube-api-access-tmnqx\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.418153 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.415207 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgq4\" (UniqueName: \"kubernetes.io/projected/1f12964a-4dd2-40d4-ae0e-53f722f5eaa1-kube-api-access-9sgq4\") pod \"ingress-canary-d4cg9\" (UID: \"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1\") " pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.419424 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.419400 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d4cg9" Jul 21 21:49:13.659437 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.659349 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:13.659621 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.659349 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:13.659621 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.659543 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:13.662240 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662215 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:49:13.662371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662302 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:49:13.662371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662343 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:49:13.662465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662365 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:49:13.662465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662368 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:49:13.662465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.662426 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:49:13.789337 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.789297 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:13.789524 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.789353 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:13.789524 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.789481 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:13.789524 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.789498 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85f4b4d747-m9xlq: secret "image-registry-tls" not found Jul 21 21:49:13.789691 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.789559 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls podName:7484c312-3831-47c9-9628-8632592ea36c nodeName:}" failed. No retries permitted until 2026-07-21 21:49:14.789541864 +0000 UTC m=+36.640301672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls") pod "image-registry-85f4b4d747-m9xlq" (UID: "7484c312-3831-47c9-9628-8632592ea36c") : secret "image-registry-tls" not found Jul 21 21:49:13.789691 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.789558 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:13.789691 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.789640 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls podName:18704d0c-cabe-4eaf-aa71-73584e07fcc6 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:14.789620448 +0000 UTC m=+36.640380274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls") pod "dns-default-w5vvr" (UID: "18704d0c-cabe-4eaf-aa71-73584e07fcc6") : secret "dns-default-metrics-tls" not found Jul 21 21:49:13.890588 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:13.890558 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:13.890792 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.890682 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:13.890792 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:13.890737 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls podName:b938b803-51a6-42c1-b29e-42c7aa84d2eb nodeName:}" failed. No retries permitted until 2026-07-21 21:49:14.890724019 +0000 UTC m=+36.741483828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8dt8" (UID: "b938b803-51a6-42c1-b29e-42c7aa84d2eb") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:14.074537 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.074498 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:14.115391 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.115352 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw"] Jul 21 21:49:14.141018 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.140984 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw"] Jul 21 21:49:14.141195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.141061 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" Jul 21 21:49:14.143549 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.143521 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tgjxk\"" Jul 21 21:49:14.143992 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.143964 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 21:49:14.144109 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.143992 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:14.163044 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.163016 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:14.216992 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.216962 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:14.292645 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.292588 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2b9\" (UniqueName: \"kubernetes.io/projected/cef6ce6a-368e-4309-8435-53804d212c25-kube-api-access-5l2b9\") pod \"migrator-7cc9c7ffd7-2fjtw\" (UID: \"cef6ce6a-368e-4309-8435-53804d212c25\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" Jul 21 21:49:14.393659 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.393568 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2b9\" (UniqueName: \"kubernetes.io/projected/cef6ce6a-368e-4309-8435-53804d212c25-kube-api-access-5l2b9\") pod \"migrator-7cc9c7ffd7-2fjtw\" (UID: \"cef6ce6a-368e-4309-8435-53804d212c25\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" Jul 21 21:49:14.402461 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.402426 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2b9\" (UniqueName: \"kubernetes.io/projected/cef6ce6a-368e-4309-8435-53804d212c25-kube-api-access-5l2b9\") pod \"migrator-7cc9c7ffd7-2fjtw\" (UID: \"cef6ce6a-368e-4309-8435-53804d212c25\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" Jul 21 21:49:14.451531 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.451377 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" Jul 21 21:49:14.572344 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.572315 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d4cg9"] Jul 21 21:49:14.589868 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.589840 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw"] Jul 21 21:49:14.663106 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:14.663005 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f12964a_4dd2_40d4_ae0e_53f722f5eaa1.slice/crio-605ec354a858a2055e88d10ab5544bdf23bf8ae5b99ba79867b435cad7912543 WatchSource:0}: Error finding container 605ec354a858a2055e88d10ab5544bdf23bf8ae5b99ba79867b435cad7912543: Status 404 returned error can't find the container with id 605ec354a858a2055e88d10ab5544bdf23bf8ae5b99ba79867b435cad7912543 Jul 21 21:49:14.663543 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:14.663513 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef6ce6a_368e_4309_8435_53804d212c25.slice/crio-8687e3a4511cfd70243a38055de15d50e19fe9ea39d02f482b5c018d0211fc82 WatchSource:0}: Error finding container 8687e3a4511cfd70243a38055de15d50e19fe9ea39d02f482b5c018d0211fc82: Status 404 returned error can't find the container with id 8687e3a4511cfd70243a38055de15d50e19fe9ea39d02f482b5c018d0211fc82 Jul 21 21:49:14.796631 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.796584 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:14.796759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.796652 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:14.796759 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.796717 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:14.796759 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.796736 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85f4b4d747-m9xlq: secret "image-registry-tls" not found Jul 21 21:49:14.796884 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.796797 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls podName:7484c312-3831-47c9-9628-8632592ea36c nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.796781879 +0000 UTC m=+38.647541691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls") pod "image-registry-85f4b4d747-m9xlq" (UID: "7484c312-3831-47c9-9628-8632592ea36c") : secret "image-registry-tls" not found Jul 21 21:49:14.796884 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.796821 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:14.796966 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.796890 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls podName:18704d0c-cabe-4eaf-aa71-73584e07fcc6 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.796873472 +0000 UTC m=+38.647633286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls") pod "dns-default-w5vvr" (UID: "18704d0c-cabe-4eaf-aa71-73584e07fcc6") : secret "dns-default-metrics-tls" not found Jul 21 21:49:14.865462 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.865429 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:14.889584 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.889541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" event={"ID":"cef6ce6a-368e-4309-8435-53804d212c25","Type":"ContainerStarted","Data":"8687e3a4511cfd70243a38055de15d50e19fe9ea39d02f482b5c018d0211fc82"} Jul 21 21:49:14.893923 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.893878 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4cg9" event={"ID":"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1","Type":"ContainerStarted","Data":"605ec354a858a2055e88d10ab5544bdf23bf8ae5b99ba79867b435cad7912543"} Jul 21 21:49:14.896433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.896408 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="490061d54b8fd1b0c86edad913977e4a6a22893550c5d6ac256eb486c3a29de8" exitCode=0 Jul 21 21:49:14.896548 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.896444 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"490061d54b8fd1b0c86edad913977e4a6a22893550c5d6ac256eb486c3a29de8"} Jul 21 21:49:14.897126 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.897108 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:14.897242 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.897224 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:14.897312 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:14.897295 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls podName:b938b803-51a6-42c1-b29e-42c7aa84d2eb nodeName:}" failed. No retries permitted until 2026-07-21 21:49:16.897279666 +0000 UTC m=+38.748039474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8dt8" (UID: "b938b803-51a6-42c1-b29e-42c7aa84d2eb") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:14.928576 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:14.928491 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:15.902974 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:15.902939 2555 generic.go:356] "Generic (PLEG): container finished" podID="6aebda04-dee6-4b4f-8434-721d2c1a5936" containerID="089120b6284214da7a67a37db262bdeba9e38e7be09c779efd6236c185fecc27" exitCode=0 Jul 21 21:49:15.903363 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:15.903019 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerDied","Data":"089120b6284214da7a67a37db262bdeba9e38e7be09c779efd6236c185fecc27"} Jul 21 21:49:16.458371 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.458335 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-76bj4"] Jul 21 21:49:16.460863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.460832 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.463568 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.463549 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 21 21:49:16.463692 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.463590 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mgcvw\"" Jul 21 21:49:16.463754 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.463709 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 21 21:49:16.464732 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.464708 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 21 21:49:16.464855 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.464741 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 21 21:49:16.470065 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.470009 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-76bj4"] Jul 21 21:49:16.499369 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.499332 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:16.555593 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.555560 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:16.610989 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.610952 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9729\" (UniqueName: \"kubernetes.io/projected/360b7c24-b314-40f9-b2b8-ec0de347c0e7-kube-api-access-h9729\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.611167 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.611018 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-cabundle\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.611167 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.611109 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-key\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.712556 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.712468 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-key\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.712556 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.712529 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9729\" (UniqueName: \"kubernetes.io/projected/360b7c24-b314-40f9-b2b8-ec0de347c0e7-kube-api-access-h9729\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.712804 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.712583 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-cabundle\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.713397 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.713374 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-cabundle\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.717375 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.717348 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/360b7c24-b314-40f9-b2b8-ec0de347c0e7-signing-key\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.724636 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.724596 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9729\" (UniqueName: \"kubernetes.io/projected/360b7c24-b314-40f9-b2b8-ec0de347c0e7-kube-api-access-h9729\") pod \"service-ca-58598695f7-76bj4\" (UID: \"360b7c24-b314-40f9-b2b8-ec0de347c0e7\") " pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.771429 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.771395 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-76bj4" Jul 21 21:49:16.813660 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.813625 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:16.813660 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.813667 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:16.813898 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.813791 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:16.813898 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.813811 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85f4b4d747-m9xlq: secret "image-registry-tls" not found Jul 21 21:49:16.813898 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.813828 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:16.813898 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.813878 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls podName:7484c312-3831-47c9-9628-8632592ea36c nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.813855956 +0000 UTC m=+42.664615768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls") pod "image-registry-85f4b4d747-m9xlq" (UID: "7484c312-3831-47c9-9628-8632592ea36c") : secret "image-registry-tls" not found Jul 21 21:49:16.813898 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.813896 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls podName:18704d0c-cabe-4eaf-aa71-73584e07fcc6 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.813887398 +0000 UTC m=+42.664647212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls") pod "dns-default-w5vvr" (UID: "18704d0c-cabe-4eaf-aa71-73584e07fcc6") : secret "dns-default-metrics-tls" not found Jul 21 21:49:16.912711 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.912666 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" event={"ID":"6aebda04-dee6-4b4f-8434-721d2c1a5936","Type":"ContainerStarted","Data":"39c285accf0eb1015ea363b7a3693d0308a9a5211dcd0a49f614fcb0fb396a80"} Jul 21 21:49:16.914182 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.914154 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:16.914317 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.914280 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:16.914667 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:16.914648 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls podName:b938b803-51a6-42c1-b29e-42c7aa84d2eb nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.914331195 +0000 UTC m=+42.765091058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8dt8" (UID: "b938b803-51a6-42c1-b29e-42c7aa84d2eb") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:16.919154 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.919118 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" event={"ID":"cef6ce6a-368e-4309-8435-53804d212c25","Type":"ContainerStarted","Data":"f3fca6f9e540d2c63f94c2778127d92a16953920f90b357587627ae922da6129"} Jul 21 21:49:16.919253 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.919206 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" event={"ID":"cef6ce6a-368e-4309-8435-53804d212c25","Type":"ContainerStarted","Data":"938a1da1fb63597262c1ccda84be98d90ec560a1d36e6b71ed3db88ddfe0751e"} Jul 21 21:49:16.942407 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.942364 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2n7nf" podStartSLOduration=7.454213355 podStartE2EDuration="37.942350728s" totalImagesPullingTime="30.488137373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:16.940718286 +0000 UTC m=+38.791478118" watchObservedRunningTime="2026-07-21 21:49:16.942350728 +0000 UTC m=+38.793110558" Jul 21 21:49:16.959959 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:16.959911 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2fjtw" podStartSLOduration=1.80549041 podStartE2EDuration="2.959898569s" totalImagesPullingTime="1.154408159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:16.9587913 +0000 UTC m=+38.809551130" watchObservedRunningTime="2026-07-21 21:49:16.959898569 +0000 UTC m=+38.810658377" Jul 21 21:49:17.083529 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.083357 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:17.101297 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.101265 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-76bj4"] Jul 21 21:49:17.104617 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:17.104577 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360b7c24_b314_40f9_b2b8_ec0de347c0e7.slice/crio-b8a933fa27a5b17c0142e9383da8a0e8b7b6176d8ad532799e76b9655bc690a8 WatchSource:0}: Error finding container b8a933fa27a5b17c0142e9383da8a0e8b7b6176d8ad532799e76b9655bc690a8: Status 404 returned error can't find the container with id b8a933fa27a5b17c0142e9383da8a0e8b7b6176d8ad532799e76b9655bc690a8 Jul 21 21:49:17.173426 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.173387 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:17.227524 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.227406 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:17.458522 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.458487 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:17.923338 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.923309 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d4cg9" event={"ID":"1f12964a-4dd2-40d4-ae0e-53f722f5eaa1","Type":"ContainerStarted","Data":"90e439997d5538d3543590131ccfcac4f3ea811768e5db93ac44c3ab2efe1089"} Jul 21 21:49:17.924626 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.924588 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-76bj4" event={"ID":"360b7c24-b314-40f9-b2b8-ec0de347c0e7","Type":"ContainerStarted","Data":"b8a933fa27a5b17c0142e9383da8a0e8b7b6176d8ad532799e76b9655bc690a8"} Jul 21 21:49:17.941507 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:17.941451 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d4cg9" podStartSLOduration=2.6343913949999997 podStartE2EDuration="4.941433644s" totalImagesPullingTime="2.307042249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:17.940176852 +0000 UTC m=+39.790936684" watchObservedRunningTime="2026-07-21 21:49:17.941433644 +0000 UTC m=+39.792193475" Jul 21 21:49:18.017363 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:18.017327 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:18.119450 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:18.119415 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:18.219000 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:18.218912 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:19.930935 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:19.930899 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-76bj4" event={"ID":"360b7c24-b314-40f9-b2b8-ec0de347c0e7","Type":"ContainerStarted","Data":"c675f4b181d984838ecec4a4410d92faa7594791203fac53d7f64cb69b2bbdc9"} Jul 21 21:49:19.949757 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:19.949699 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-76bj4" podStartSLOduration=1.992371754 podStartE2EDuration="3.949682438s" totalImagesPullingTime="1.957310684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:19.949051962 +0000 UTC m=+41.799811797" watchObservedRunningTime="2026-07-21 21:49:19.949682438 +0000 UTC m=+41.800442268" Jul 21 21:49:20.194117 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.194024 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.382255 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.382219 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.482561 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.482471 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.617180 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.617141 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.733989 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.733876 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.819946 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.819899 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.846265 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.846226 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:20.846452 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.846281 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:20.846452 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.846426 2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:20.846652 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.846503 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls podName:18704d0c-cabe-4eaf-aa71-73584e07fcc6 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:28.846481624 +0000 UTC m=+50.697241438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls") pod "dns-default-w5vvr" (UID: "18704d0c-cabe-4eaf-aa71-73584e07fcc6") : secret "dns-default-metrics-tls" not found Jul 21 21:49:20.846938 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.846909 2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:20.846938 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.846931 2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85f4b4d747-m9xlq: secret "image-registry-tls" not found Jul 21 21:49:20.847117 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.846998 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls podName:7484c312-3831-47c9-9628-8632592ea36c nodeName:}" failed. No retries permitted until 2026-07-21 21:49:28.846982108 +0000 UTC m=+50.697741921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls") pod "image-registry-85f4b4d747-m9xlq" (UID: "7484c312-3831-47c9-9628-8632592ea36c") : secret "image-registry-tls" not found Jul 21 21:49:20.918062 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.918013 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:20.946996 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:20.946961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:20.947432 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.947121 2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:20.947432 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:20.947191 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls podName:b938b803-51a6-42c1-b29e-42c7aa84d2eb nodeName:}" failed. No retries permitted until 2026-07-21 21:49:28.947171279 +0000 UTC m=+50.797931094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8dt8" (UID: "b938b803-51a6-42c1-b29e-42c7aa84d2eb") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:21.507665 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:21.507599 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:22.294363 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:22.294328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:22.348075 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:22.348036 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:22.702487 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:22.702453 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:22.889197 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:22.889159 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:23.044058 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:23.043963 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:23.099216 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:23.099167 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:23.495494 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:23.495462 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:23.892977 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:23.892947 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:23.996548 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:23.996519 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:24.097370 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:24.097339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:24.688399 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:24.688363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:24.894525 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:24.894493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:24.947794 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:24.947721 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:25.292419 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:25.292344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:25.498024 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:25.497987 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:25.695591 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:25.695554 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:26.526669 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:26.526639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:26.694652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:26.694542 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:27.091737 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:27.091700 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:27.296725 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:27.296692 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:27.499069 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:27.498992 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:27.705368 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:27.705339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:28.898018 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.897988 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:28.910106 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.910077 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:28.910210 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.910112 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:28.913264 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.913236 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18704d0c-cabe-4eaf-aa71-73584e07fcc6-metrics-tls\") pod \"dns-default-w5vvr\" (UID: \"18704d0c-cabe-4eaf-aa71-73584e07fcc6\") " pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:28.913358 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.913345 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"image-registry-85f4b4d747-m9xlq\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:28.986508 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.986468 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:28.999735 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:28.999708 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:29.011440 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.011409 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:29.013658 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.013640 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b938b803-51a6-42c1-b29e-42c7aa84d2eb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8dt8\" (UID: \"b938b803-51a6-42c1-b29e-42c7aa84d2eb\") " pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:29.038508 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.038482 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8dt8" Jul 21 21:49:29.097362 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.096599 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.174214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.174176 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:49:29.177905 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:29.177881 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7484c312_3831_47c9_9628_8632592ea36c.slice/crio-4da3db89d6d827d6c896a83df1794dcfa90ff042d3a51dc1cc18e86f968d9b5c WatchSource:0}: Error finding container 4da3db89d6d827d6c896a83df1794dcfa90ff042d3a51dc1cc18e86f968d9b5c: Status 404 returned error can't find the container with id 4da3db89d6d827d6c896a83df1794dcfa90ff042d3a51dc1cc18e86f968d9b5c Jul 21 21:49:29.187634 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.187246 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w5vvr"] Jul 21 21:49:29.189457 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:29.189433 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18704d0c_cabe_4eaf_aa71_73584e07fcc6.slice/crio-c07c9e2f10e5187054cc20fb7c0618b44d4a10d4af15c97bca6afcc687f93d67 WatchSource:0}: Error finding container c07c9e2f10e5187054cc20fb7c0618b44d4a10d4af15c97bca6afcc687f93d67: Status 404 returned error can't find the container with id c07c9e2f10e5187054cc20fb7c0618b44d4a10d4af15c97bca6afcc687f93d67 Jul 21 21:49:29.223021 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.222993 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8dt8"] Jul 21 21:49:29.228530 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:29.228508 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb938b803_51a6_42c1_b29e_42c7aa84d2eb.slice/crio-2966bd241e3135f06199f5e6cf5efa137904cef61a23f479c0a868c320360e79 WatchSource:0}: Error finding container 2966bd241e3135f06199f5e6cf5efa137904cef61a23f479c0a868c320360e79: Status 404 returned error can't find the container with id 2966bd241e3135f06199f5e6cf5efa137904cef61a23f479c0a868c320360e79 Jul 21 21:49:29.290069 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.290041 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.511686 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.511591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.682147 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.682107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.734031 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.734002 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.893793 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.893755 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:29.958154 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.957371 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" event={"ID":"7484c312-3831-47c9-9628-8632592ea36c","Type":"ContainerStarted","Data":"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9"} Jul 21 21:49:29.958154 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.957435 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" event={"ID":"7484c312-3831-47c9-9628-8632592ea36c","Type":"ContainerStarted","Data":"4da3db89d6d827d6c896a83df1794dcfa90ff042d3a51dc1cc18e86f968d9b5c"} Jul 21 21:49:29.958154 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.957750 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:29.959795 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.959722 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8dt8" event={"ID":"b938b803-51a6-42c1-b29e-42c7aa84d2eb","Type":"ContainerStarted","Data":"2966bd241e3135f06199f5e6cf5efa137904cef61a23f479c0a868c320360e79"} Jul 21 21:49:29.962977 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:29.962917 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5vvr" event={"ID":"18704d0c-cabe-4eaf-aa71-73584e07fcc6","Type":"ContainerStarted","Data":"c07c9e2f10e5187054cc20fb7c0618b44d4a10d4af15c97bca6afcc687f93d67"} Jul 21 21:49:30.102000 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.101962 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.275851 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.275770 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.328184 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.328124 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.498051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.498013 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.717083 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.717051 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.771116 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.771073 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:30.895296 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:30.895231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:31.110720 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:31.110687 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:31.161023 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:31.160985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:31.296584 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:31.296550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:31.696530 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:31.696481 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:31.972123 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:31.972034 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5vvr" event={"ID":"18704d0c-cabe-4eaf-aa71-73584e07fcc6","Type":"ContainerStarted","Data":"95cf5c89919ddfeabbf80b0869139514af4e6076e39f844d3fee6cfe23f4bcec"} Jul 21 21:49:32.624071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:32.624016 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:33.019060 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:33.018978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:33.119022 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:33.118971 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:33.173481 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:33.173447 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:35.987374 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:35.987340 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w5vvr" event={"ID":"18704d0c-cabe-4eaf-aa71-73584e07fcc6","Type":"ContainerStarted","Data":"8777391cafc168e1ae1c9ff5cfc9a2b43c079e354bfdca2c15273f0d0a7640bd"} Jul 21 21:49:35.987829 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:35.987449 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:35.988701 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:35.988678 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8dt8" event={"ID":"b938b803-51a6-42c1-b29e-42c7aa84d2eb","Type":"ContainerStarted","Data":"d14076d75b432c11ec0bab67efed1314d0660661b478f5157b5eb8f5c7cab145"} Jul 21 21:49:36.006099 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:36.006059 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w5vvr" podStartSLOduration=20.521202438 podStartE2EDuration="23.006048059s" totalImagesPullingTime="2.484845621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:36.004799393 +0000 UTC m=+57.855559227" watchObservedRunningTime="2026-07-21 21:49:36.006048059 +0000 UTC m=+57.856807890" Jul 21 21:49:36.006383 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:36.006362 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" podStartSLOduration=32.006355292 podStartE2EDuration="32.006355292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:29.982839163 +0000 UTC m=+51.833598998" watchObservedRunningTime="2026-07-21 21:49:36.006355292 +0000 UTC m=+57.857115100" Jul 21 21:49:37.730195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.730146 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4"] Jul 21 21:49:37.738759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.737369 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp"] Jul 21 21:49:37.738759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.737479 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.740583 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.740532 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 21:49:37.740745 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.740711 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qnvgn\"" Jul 21 21:49:37.741367 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.741051 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 21:49:37.752296 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.752271 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2"] Jul 21 21:49:37.752423 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.752402 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.754336 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.754317 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.755069 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.754816 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j"] Jul 21 21:49:37.755262 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.755232 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 21:49:37.756273 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756252 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 21:49:37.756581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756513 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4"] Jul 21 21:49:37.756581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756553 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp"] Jul 21 21:49:37.756581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756565 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 21:49:37.756581 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756579 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:37.757435 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.756781 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 21:49:37.759640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.759598 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 21 21:49:37.759945 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.759753 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 21 21:49:37.760015 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.759959 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 21 21:49:37.760015 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.759995 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 21 21:49:37.760501 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.760242 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-v88lx\"" Jul 21 21:49:37.760501 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.760322 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 21 21:49:37.761080 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.761054 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j"] Jul 21 21:49:37.770677 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.770638 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2"] Jul 21 21:49:37.772012 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.771981 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:49:37.786703 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.786664 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.786825 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.786724 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.831488 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.831460 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68c56456cb-xcdbv"] Jul 21 21:49:37.834403 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.834376 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.859463 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.859436 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c56456cb-xcdbv"] Jul 21 21:49:37.866007 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.865983 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:37.887281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.887443 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887290 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-registry-certificates\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887443 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887396 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7xk\" (UniqueName: \"kubernetes.io/projected/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-kube-api-access-fc7xk\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.887443 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887429 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-installation-pull-secrets\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887459 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crs5\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-kube-api-access-6crs5\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887485 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf776\" (UniqueName: \"kubernetes.io/projected/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-kube-api-access-zf776\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.887582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887555 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.887736 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-registry-tls\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887736 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887660 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-image-registry-private-configuration\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887736 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887722 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.887863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887753 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-trusted-ca\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887782 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-bound-sa-token\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.887863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887818 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7blw\" (UniqueName: \"kubernetes.io/projected/28aa3d7d-4493-4933-bdb0-eb749b5174d8-kube-api-access-k7blw\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:37.887863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887847 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.888004 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887873 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.888004 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887899 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57046bca-034f-4450-aa04-d5c7572b71d3-ca-trust-extracted\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.888004 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887940 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.888004 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.887989 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-tmp\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.888165 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.888138 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.888228 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.888213 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.888279 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.888241 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28aa3d7d-4493-4933-bdb0-eb749b5174d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:37.888492 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.888467 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.890812 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.890768 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5fa6eff-8f54-4c83-a2ac-a5006592dfa2-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4bcf4\" (UID: \"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:37.943063 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.943018 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:37.989372 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989280 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-tmp\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.989372 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989342 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.989640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989496 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.989640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989526 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28aa3d7d-4493-4933-bdb0-eb749b5174d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:37.989640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989595 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.989640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989638 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-registry-certificates\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989671 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7xk\" (UniqueName: \"kubernetes.io/projected/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-kube-api-access-fc7xk\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989695 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-installation-pull-secrets\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989736 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crs5\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-kube-api-access-6crs5\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989754 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-tmp\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989776 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf776\" (UniqueName: \"kubernetes.io/projected/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-kube-api-access-zf776\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989827 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:37.989867 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.989866 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-registry-tls\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.990898 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.990804 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-image-registry-private-configuration\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.992806 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.992773 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-registry-certificates\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.994742 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.994710 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-installation-pull-secrets\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.994964 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.992782 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-trusted-ca\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.995115 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.995100 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-bound-sa-token\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.995257 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.995239 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7blw\" (UniqueName: \"kubernetes.io/projected/28aa3d7d-4493-4933-bdb0-eb749b5174d8-kube-api-access-k7blw\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:37.995381 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.995366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.995738 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.995717 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.996010 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.995978 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57046bca-034f-4450-aa04-d5c7572b71d3-ca-trust-extracted\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.996136 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.996031 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.997298 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.997056 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57046bca-034f-4450-aa04-d5c7572b71d3-ca-trust-extracted\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:37.999117 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.997920 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:37.999117 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.999056 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28aa3d7d-4493-4933-bdb0-eb749b5174d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:38.000480 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:37.999293 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-ca\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:38.000480 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.000009 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:38.003728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.002360 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57046bca-034f-4450-aa04-d5c7572b71d3-trusted-ca\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:38.003728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.003252 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57046bca-034f-4450-aa04-d5c7572b71d3-image-registry-private-configuration\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:38.003728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.003425 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-registry-tls\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:38.003728 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.003697 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crs5\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-kube-api-access-6crs5\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:38.003964 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.003765 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:38.004193 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.004167 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf776\" (UniqueName: \"kubernetes.io/projected/89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf-kube-api-access-zf776\") pod \"klusterlet-addon-workmgr-598f86c444-snkrp\" (UID: \"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:38.006062 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.006035 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:38.006816 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.006791 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7xk\" (UniqueName: \"kubernetes.io/projected/890b7ed0-ee19-4fe3-a29c-1068daf7aac5-kube-api-access-fc7xk\") pod \"cluster-proxy-proxy-agent-6b9574fc8f-2xdj2\" (UID: \"890b7ed0-ee19-4fe3-a29c-1068daf7aac5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:38.013036 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.013003 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57046bca-034f-4450-aa04-d5c7572b71d3-bound-sa-token\") pod \"image-registry-68c56456cb-xcdbv\" (UID: \"57046bca-034f-4450-aa04-d5c7572b71d3\") " pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:38.013341 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.013318 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7blw\" (UniqueName: \"kubernetes.io/projected/28aa3d7d-4493-4933-bdb0-eb749b5174d8-kube-api-access-k7blw\") pod \"managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j\" (UID: \"28aa3d7d-4493-4933-bdb0-eb749b5174d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:38.044399 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.044363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:38.049146 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.049117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" Jul 21 21:49:38.066762 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.066738 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:38.082759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.082735 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" Jul 21 21:49:38.089427 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.089404 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" Jul 21 21:49:38.143996 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.143964 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:38.155095 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:38.155069 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:39.227878 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.227845 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:39.287487 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.287458 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:39.492562 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.491503 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c56456cb-xcdbv"] Jul 21 21:49:39.516311 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.516288 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp"] Jul 21 21:49:39.519108 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.519082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2"] Jul 21 21:49:39.523992 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:39.523967 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89baabb1_e8a0_4c2b_b22e_918ae3f5f7cf.slice/crio-65761262e1bce61fb89b2728b7a5d5fed1d782bbb75a3b53de5d2ec064c31302 WatchSource:0}: Error finding container 65761262e1bce61fb89b2728b7a5d5fed1d782bbb75a3b53de5d2ec064c31302: Status 404 returned error can't find the container with id 65761262e1bce61fb89b2728b7a5d5fed1d782bbb75a3b53de5d2ec064c31302 Jul 21 21:49:39.524204 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:39.524180 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod890b7ed0_ee19_4fe3_a29c_1068daf7aac5.slice/crio-1ced617609458d1a62490b5613f69441806b5aa7877ce9ec99c4b51be2996d0e WatchSource:0}: Error finding container 1ced617609458d1a62490b5613f69441806b5aa7877ce9ec99c4b51be2996d0e: Status 404 returned error can't find the container with id 1ced617609458d1a62490b5613f69441806b5aa7877ce9ec99c4b51be2996d0e Jul 21 21:49:39.540662 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.540638 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j"] Jul 21 21:49:39.543085 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:39.543065 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28aa3d7d_4493_4933_bdb0_eb749b5174d8.slice/crio-6d15b02d95dec30c7fb7a4c037731ae66b066e6c8f8a365e27b827b487549c5d WatchSource:0}: Error finding container 6d15b02d95dec30c7fb7a4c037731ae66b066e6c8f8a365e27b827b487549c5d: Status 404 returned error can't find the container with id 6d15b02d95dec30c7fb7a4c037731ae66b066e6c8f8a365e27b827b487549c5d Jul 21 21:49:39.705131 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:39.705099 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4"] Jul 21 21:49:39.708710 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:39.708681 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fa6eff_8f54_4c83_a2ac_a5006592dfa2.slice/crio-095f6bfd9ede5c7cd58ada7a66247a09f25a6a7b8d93812c93f94119084e2830 WatchSource:0}: Error finding container 095f6bfd9ede5c7cd58ada7a66247a09f25a6a7b8d93812c93f94119084e2830: Status 404 returned error can't find the container with id 095f6bfd9ede5c7cd58ada7a66247a09f25a6a7b8d93812c93f94119084e2830 Jul 21 21:49:40.013808 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.013562 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" event={"ID":"57046bca-034f-4450-aa04-d5c7572b71d3","Type":"ContainerStarted","Data":"ac0e34e32fa82639b979ab3b33d891b90b81ea530da266b69cf710f1a70de1a3"} Jul 21 21:49:40.013808 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.013643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" event={"ID":"57046bca-034f-4450-aa04-d5c7572b71d3","Type":"ContainerStarted","Data":"dfeb092088e641ceedb82f3c0e14f51208cb86a22b0479cbe7016f11f7e9c3da"} Jul 21 21:49:40.013808 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.013676 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:49:40.015805 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.015775 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8dt8" event={"ID":"b938b803-51a6-42c1-b29e-42c7aa84d2eb","Type":"ContainerStarted","Data":"26cb9e9373cef697b496e7c2cd41ad14ae0bd7f7031eaf5d61c7f4f3e6d25851"} Jul 21 21:49:40.017075 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.017048 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" event={"ID":"28aa3d7d-4493-4933-bdb0-eb749b5174d8","Type":"ContainerStarted","Data":"6d15b02d95dec30c7fb7a4c037731ae66b066e6c8f8a365e27b827b487549c5d"} Jul 21 21:49:40.018357 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.018335 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" event={"ID":"890b7ed0-ee19-4fe3-a29c-1068daf7aac5","Type":"ContainerStarted","Data":"1ced617609458d1a62490b5613f69441806b5aa7877ce9ec99c4b51be2996d0e"} Jul 21 21:49:40.019720 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.019690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" event={"ID":"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2","Type":"ContainerStarted","Data":"095f6bfd9ede5c7cd58ada7a66247a09f25a6a7b8d93812c93f94119084e2830"} Jul 21 21:49:40.020817 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.020793 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" event={"ID":"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf","Type":"ContainerStarted","Data":"65761262e1bce61fb89b2728b7a5d5fed1d782bbb75a3b53de5d2ec064c31302"} Jul 21 21:49:40.033733 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.033686 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" podStartSLOduration=3.033672073 podStartE2EDuration="3.033672073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:40.032583237 +0000 UTC m=+61.883343069" watchObservedRunningTime="2026-07-21 21:49:40.033672073 +0000 UTC m=+61.884431882" Jul 21 21:49:40.160384 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.160354 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9khl7" Jul 21 21:49:40.323830 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.323758 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:40.384897 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:40.384829 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:42.035456 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:42.035370 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" event={"ID":"a5fa6eff-8f54-4c83-a2ac-a5006592dfa2","Type":"ContainerStarted","Data":"dbf9fe214e447a6a99d3a06cd84131e450c49672dfd664cb150c9fc14fe63bb6"} Jul 21 21:49:42.054535 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:42.054476 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4bcf4" podStartSLOduration=3.141627822 podStartE2EDuration="5.054456293s" totalImagesPullingTime="1.912828471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:42.052808772 +0000 UTC m=+63.903568604" watchObservedRunningTime="2026-07-21 21:49:42.054456293 +0000 UTC m=+63.905216125" Jul 21 21:49:42.182979 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:42.182910 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:42.240091 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:42.239981 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:42.995347 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:42.995318 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w5vvr" Jul 21 21:49:43.159089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:43.159010 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:43.210718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:43.210686 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:43.265169 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:43.265079 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:44.462343 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.462311 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:44.462861 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.462428 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:44.465317 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.465281 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:49:44.465471 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.465398 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:49:44.475416 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.475390 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aba52102-36ce-4326-9452-0e4456e6f912-metrics-certs\") pod \"network-metrics-daemon-85x75\" (UID: \"aba52102-36ce-4326-9452-0e4456e6f912\") " pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:44.475416 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.475401 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0298deea-f91e-4d59-9170-a1914c0e4491-original-pull-secret\") pod \"global-pull-secret-syncer-tfqbs\" (UID: \"0298deea-f91e-4d59-9170-a1914c0e4491\") " pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:44.563673 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.563630 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:44.566814 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.566789 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:49:44.573655 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.573632 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:49:44.576793 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.576760 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:49:44.581464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.581441 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-85x75" Jul 21 21:49:44.585224 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.585207 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tfqbs" Jul 21 21:49:44.587226 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.587205 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlsc\" (UniqueName: \"kubernetes.io/projected/e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0-kube-api-access-xhlsc\") pod \"network-check-target-8gzkz\" (UID: \"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0\") " pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:44.882095 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.882058 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:49:44.889647 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.889590 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:44.923014 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:44.922981 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:45.023824 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:45.023792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:45.127443 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:45.127406 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:45.227514 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:45.227416 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:45.292116 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:45.292087 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:45.614220 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:45.614186 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:47.777792 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:47.777758 2555 patch_prober.go:28] interesting pod/image-registry-85f4b4d747-m9xlq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 21:49:47.778165 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:47.777822 2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" podUID="7484c312-3831-47c9-9628-8632592ea36c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:49:48.108677 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.108449 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gzkz"] Jul 21 21:49:48.114014 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:48.113974 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3055aa8_b3b2_459f_88e9_ec4ea7c6bdb0.slice/crio-0634472096f9437448051d8c536b6b25226a3ea62f889532f2a4e25201c64358 WatchSource:0}: Error finding container 0634472096f9437448051d8c536b6b25226a3ea62f889532f2a4e25201c64358: Status 404 returned error can't find the container with id 0634472096f9437448051d8c536b6b25226a3ea62f889532f2a4e25201c64358 Jul 21 21:49:48.202863 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.202831 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:48.304515 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.304488 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:48.336469 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.336446 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tfqbs"] Jul 21 21:49:48.339519 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:48.339498 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0298deea_f91e_4d59_9170_a1914c0e4491.slice/crio-6764129038d60089661a314428c5288c99487f4a0e3d53861786354574ce03b2 WatchSource:0}: Error finding container 6764129038d60089661a314428c5288c99487f4a0e3d53861786354574ce03b2: Status 404 returned error can't find the container with id 6764129038d60089661a314428c5288c99487f4a0e3d53861786354574ce03b2 Jul 21 21:49:48.345085 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.345062 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-85x75"] Jul 21 21:49:48.347860 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:49:48.347840 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba52102_36ce_4326_9452_0e4456e6f912.slice/crio-dcfd27ffe2e05fc9ed3e39d1c677d53621c10c92b8e4bc53f4c16abb603b3163 WatchSource:0}: Error finding container dcfd27ffe2e05fc9ed3e39d1c677d53621c10c92b8e4bc53f4c16abb603b3163: Status 404 returned error can't find the container with id dcfd27ffe2e05fc9ed3e39d1c677d53621c10c92b8e4bc53f4c16abb603b3163 Jul 21 21:49:48.402991 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:48.402949 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:49.066564 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.066526 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gzkz" event={"ID":"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0","Type":"ContainerStarted","Data":"0634472096f9437448051d8c536b6b25226a3ea62f889532f2a4e25201c64358"} Jul 21 21:49:49.069994 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.069237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8dt8" event={"ID":"b938b803-51a6-42c1-b29e-42c7aa84d2eb","Type":"ContainerStarted","Data":"86f041c7b7c11c3f1f72b9e9d48c771d1f87c2446af16f16c645115bfeebac81"} Jul 21 21:49:49.072205 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.071733 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" event={"ID":"28aa3d7d-4493-4933-bdb0-eb749b5174d8","Type":"ContainerStarted","Data":"9c6a833af4ef7dfb20f76f6d7523ab438cf0a93285b6dcd790036438bc633975"} Jul 21 21:49:49.078038 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.078006 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" event={"ID":"890b7ed0-ee19-4fe3-a29c-1068daf7aac5","Type":"ContainerStarted","Data":"bd9de54b682d5d9f0aba3291fc1f6e1d2344a94a6e2dc52cbaff8fe94bbce880"} Jul 21 21:49:49.078189 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.078054 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" event={"ID":"890b7ed0-ee19-4fe3-a29c-1068daf7aac5","Type":"ContainerStarted","Data":"923c89a9e370bc460035f6f218c84a76a989d98817890cfb3738418909d5e36a"} Jul 21 21:49:49.078189 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.078073 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" event={"ID":"890b7ed0-ee19-4fe3-a29c-1068daf7aac5","Type":"ContainerStarted","Data":"fb67039fe73c4ece7543ac67761d07a49321fb8cae76dd0b097a7763515186b0"} Jul 21 21:49:49.079470 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.079443 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tfqbs" event={"ID":"0298deea-f91e-4d59-9170-a1914c0e4491","Type":"ContainerStarted","Data":"6764129038d60089661a314428c5288c99487f4a0e3d53861786354574ce03b2"} Jul 21 21:49:49.081198 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.081172 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85x75" event={"ID":"aba52102-36ce-4326-9452-0e4456e6f912","Type":"ContainerStarted","Data":"dcfd27ffe2e05fc9ed3e39d1c677d53621c10c92b8e4bc53f4c16abb603b3163"} Jul 21 21:49:49.083652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.083148 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" event={"ID":"89baabb1-e8a0-4c2b-b22e-918ae3f5f7cf","Type":"ContainerStarted","Data":"82942c4657e908cb386d622f37cd08904ff7e0768c804e88e43a24a785a7f14a"} Jul 21 21:49:49.083652 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.083639 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:49.085739 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.085717 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" Jul 21 21:49:49.089725 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.089674 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w8dt8" podStartSLOduration=17.572232815 podStartE2EDuration="36.089656229s" totalImagesPullingTime="18.517423414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:49.088490084 +0000 UTC m=+70.939249917" watchObservedRunningTime="2026-07-21 21:49:49.089656229 +0000 UTC m=+70.940416059" Jul 21 21:49:49.106577 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.106029 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598f86c444-snkrp" podStartSLOduration=3.690802564 podStartE2EDuration="12.106008473s" totalImagesPullingTime="8.415205909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:49.104578316 +0000 UTC m=+70.955338149" watchObservedRunningTime="2026-07-21 21:49:49.106008473 +0000 UTC m=+70.956768305" Jul 21 21:49:49.125265 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.125205 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9574fc8f-2xdj2" podStartSLOduration=3.699927699 podStartE2EDuration="12.125181416s" totalImagesPullingTime="8.425253717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:49.124310893 +0000 UTC m=+70.975070725" watchObservedRunningTime="2026-07-21 21:49:49.125181416 +0000 UTC m=+70.975941252" Jul 21 21:49:49.140849 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:49.140483 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c84dc8d8b-4hb2j" podStartSLOduration=3.745694365 podStartE2EDuration="12.140454518s" totalImagesPullingTime="8.394760153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:49.140100312 +0000 UTC m=+70.990860146" watchObservedRunningTime="2026-07-21 21:49:49.140454518 +0000 UTC m=+70.991214334" Jul 21 21:49:51.094276 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.094229 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85x75" event={"ID":"aba52102-36ce-4326-9452-0e4456e6f912","Type":"ContainerStarted","Data":"c911be35a34c68cd79bf0aee5c4497c4c5e712cd371341cdf79e07611fefe532"} Jul 21 21:49:51.094793 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.094302 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-85x75" event={"ID":"aba52102-36ce-4326-9452-0e4456e6f912","Type":"ContainerStarted","Data":"d023603a0471635d8ecd2782c674bb2801710816e5ed1caaaae2d79ec7180576"} Jul 21 21:49:51.113936 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.113867 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-85x75" podStartSLOduration=70.47574673 podStartE2EDuration="1m12.113847067s" totalImagesPullingTime="1.638100337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:51.111463244 +0000 UTC m=+72.962223079" watchObservedRunningTime="2026-07-21 21:49:51.113847067 +0000 UTC m=+72.964606901" Jul 21 21:49:51.124684 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.124639 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:51.182857 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.182821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:51.285803 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.285766 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:51.357136 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.357056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:51.414967 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.414931 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:51.977365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:51.977322 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:52.076256 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:52.076221 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:52.186597 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:52.186561 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:52.324228 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:52.324136 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:53.103819 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.103786 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tfqbs" event={"ID":"0298deea-f91e-4d59-9170-a1914c0e4491","Type":"ContainerStarted","Data":"30e74714691088856353d6796dff8b71f1e7d6eb328252ffad16b5d27451b855"} Jul 21 21:49:53.105423 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.105397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gzkz" event={"ID":"e3055aa8-b3b2-459f-88e9-ec4ea7c6bdb0","Type":"ContainerStarted","Data":"1ac71b7b1a26af5964a927f769d6f239ddecd9a543e39060945150d8f7b33aea"} Jul 21 21:49:53.105554 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.105536 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:49:53.119953 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.119904 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tfqbs" podStartSLOduration=69.555651355 podStartE2EDuration="1m14.119887009s" totalImagesPullingTime="4.564235654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:53.119080771 +0000 UTC m=+74.969840602" watchObservedRunningTime="2026-07-21 21:49:53.119887009 +0000 UTC m=+74.970646840" Jul 21 21:49:53.134929 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.134888 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gzkz" podStartSLOduration=70.349933651 podStartE2EDuration="1m15.134872609s" totalImagesPullingTime="4.784938958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:53.134014973 +0000 UTC m=+74.984774805" watchObservedRunningTime="2026-07-21 21:49:53.134872609 +0000 UTC m=+74.985632440" Jul 21 21:49:53.305578 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.305490 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:53.409468 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.409435 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:53.507664 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:53.507631 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:56.289650 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:56.289616 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:56.388788 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:56.388756 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:56.488223 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:56.488192 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:57.161575 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:57.161540 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:57.280179 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:57.280144 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:57.359534 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:57.359501 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:57.776362 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:57.776337 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:49:58.163959 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:58.163933 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:59.486565 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.486529 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:59.669036 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.669004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:59.745907 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.745821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:59.781536 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.781501 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fhssk"] Jul 21 21:49:59.784842 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.784809 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.788982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.788962 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 21:49:59.789503 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.789477 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7blhv\"" Jul 21 21:49:59.789503 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.789491 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 21:49:59.790288 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.790267 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 21:49:59.790288 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.790284 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 21:49:59.790438 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.790289 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 21:49:59.790438 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.790266 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 21:49:59.879464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.879357 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:59.886389 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886359 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-metrics-client-ca\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886512 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886399 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886512 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886431 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-textfile\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886512 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886500 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-root\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886687 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886559 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886687 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886585 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-wtmp\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886687 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886650 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd2n\" (UniqueName: \"kubernetes.io/projected/8043d00d-884b-4252-9c24-fb2ec17f1a70-kube-api-access-rnd2n\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886687 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886683 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.886864 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.886709 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-sys\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.987981 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.987950 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-root\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.987998 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988027 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-wtmp\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd2n\" (UniqueName: \"kubernetes.io/projected/8043d00d-884b-4252-9c24-fb2ec17f1a70-kube-api-access-rnd2n\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988069 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-root\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988091 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:59.988157 2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 21:49:59.988200 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988184 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-sys\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988210 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-wtmp\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:49:59.988232 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls podName:8043d00d-884b-4252-9c24-fb2ec17f1a70 nodeName:}" failed. No retries permitted until 2026-07-21 21:50:00.488208948 +0000 UTC m=+82.338968759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls") pod "node-exporter-fhssk" (UID: "8043d00d-884b-4252-9c24-fb2ec17f1a70") : secret "node-exporter-tls" not found Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988244 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8043d00d-884b-4252-9c24-fb2ec17f1a70-sys\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988304 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-metrics-client-ca\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988333 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988542 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988369 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-textfile\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988806 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988759 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-textfile\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.988968 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988949 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.989020 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.988988 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8043d00d-884b-4252-9c24-fb2ec17f1a70-metrics-client-ca\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:49:59.990759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:49:59.990736 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:50:00.034561 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.034477 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd2n\" (UniqueName: \"kubernetes.io/projected/8043d00d-884b-4252-9c24-fb2ec17f1a70-kube-api-access-rnd2n\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:50:00.063905 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.063873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.162856 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.162820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.259644 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.259614 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.359270 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.359240 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.459666 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.459635 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.494078 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.494046 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:50:00.496299 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.496273 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8043d00d-884b-4252-9c24-fb2ec17f1a70-node-exporter-tls\") pod \"node-exporter-fhssk\" (UID: \"8043d00d-884b-4252-9c24-fb2ec17f1a70\") " pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:50:00.654849 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.654764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.696338 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.696309 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhssk" Jul 21 21:50:00.707559 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:50:00.707532 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8043d00d_884b_4252_9c24_fb2ec17f1a70.slice/crio-d74c78c935617755a688ed99e491cf8a620a198e87fe65e7841d5f51cf059389 WatchSource:0}: Error finding container d74c78c935617755a688ed99e491cf8a620a198e87fe65e7841d5f51cf059389: Status 404 returned error can't find the container with id d74c78c935617755a688ed99e491cf8a620a198e87fe65e7841d5f51cf059389 Jul 21 21:50:00.735415 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.735393 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:00.997989 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:00.997916 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:01.030493 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:01.030467 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68c56456cb-xcdbv" Jul 21 21:50:01.134506 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:01.134473 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhssk" event={"ID":"8043d00d-884b-4252-9c24-fb2ec17f1a70","Type":"ContainerStarted","Data":"d74c78c935617755a688ed99e491cf8a620a198e87fe65e7841d5f51cf059389"} Jul 21 21:50:01.163586 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:01.163549 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:01.793654 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:01.793590 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:01.859008 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:01.858904 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:02.140100 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:02.140026 2555 generic.go:356] "Generic (PLEG): container finished" podID="8043d00d-884b-4252-9c24-fb2ec17f1a70" containerID="f7e8b3f3e801655dee9f7725400481aaa752e9ea3938ec4f4d7ef25a5d1a1b41" exitCode=0 Jul 21 21:50:02.140100 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:02.140082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhssk" event={"ID":"8043d00d-884b-4252-9c24-fb2ec17f1a70","Type":"ContainerDied","Data":"f7e8b3f3e801655dee9f7725400481aaa752e9ea3938ec4f4d7ef25a5d1a1b41"} Jul 21 21:50:02.688590 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:02.688557 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:02.747632 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:02.747578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:02.795561 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:02.795523 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" podUID="7484c312-3831-47c9-9628-8632592ea36c" containerName="registry" containerID="cri-o://4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9" gracePeriod=30 Jul 21 21:50:03.029976 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.029941 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:03.072760 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.072735 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:50:03.103205 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.103178 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:03.144579 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.144549 2555 generic.go:356] "Generic (PLEG): container finished" podID="7484c312-3831-47c9-9628-8632592ea36c" containerID="4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9" exitCode=0 Jul 21 21:50:03.144758 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.144601 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" Jul 21 21:50:03.144758 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.144643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" event={"ID":"7484c312-3831-47c9-9628-8632592ea36c","Type":"ContainerDied","Data":"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9"} Jul 21 21:50:03.144758 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.144690 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85f4b4d747-m9xlq" event={"ID":"7484c312-3831-47c9-9628-8632592ea36c","Type":"ContainerDied","Data":"4da3db89d6d827d6c896a83df1794dcfa90ff042d3a51dc1cc18e86f968d9b5c"} Jul 21 21:50:03.144758 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.144718 2555 scope.go:117] "RemoveContainer" containerID="4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9" Jul 21 21:50:03.147036 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.147015 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhssk" event={"ID":"8043d00d-884b-4252-9c24-fb2ec17f1a70","Type":"ContainerStarted","Data":"ae3661244d57a72aa2069a9311b7df6182ec3e9d8512b396d2032a4b2804976e"} Jul 21 21:50:03.147142 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.147065 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhssk" event={"ID":"8043d00d-884b-4252-9c24-fb2ec17f1a70","Type":"ContainerStarted","Data":"d515ce66365780bf3109adef421a97a5afb0aa652915c183592814443a7eeaf9"} Jul 21 21:50:03.153037 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.153017 2555 scope.go:117] "RemoveContainer" containerID="4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9" Jul 21 21:50:03.153270 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:50:03.153252 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9\": container with ID starting with 4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9 not found: ID does not exist" containerID="4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9" Jul 21 21:50:03.153333 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.153277 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9"} err="failed to get container status \"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9\": rpc error: code = NotFound desc = could not find container \"4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9\": container with ID starting with 4faf023cfa145d153815a145619708d55765265bf8c5f0b18f8bc86d54e955f9 not found: ID does not exist" Jul 21 21:50:03.159718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.159698 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:03.170996 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.170954 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fhssk" podStartSLOduration=3.23571681 podStartE2EDuration="4.17094008s" totalImagesPullingTime="872.023619ms" totalInitContainerRuntime="63.199651ms" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:03.170211868 +0000 UTC m=+85.020971701" watchObservedRunningTime="2026-07-21 21:50:03.17094008 +0000 UTC m=+85.021699911" Jul 21 21:50:03.216100 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216073 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216122 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216164 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216189 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216330 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216236 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216330 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216270 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216429 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216378 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216481 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216433 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rddr\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr\") pod \"7484c312-3831-47c9-9628-8632592ea36c\" (UID: \"7484c312-3831-47c9-9628-8632592ea36c\") " Jul 21 21:50:03.216874 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216832 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:03.216972 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.216884 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:03.219235 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.219213 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:03.219369 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.219344 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr" (OuterVolumeSpecName: "kube-api-access-7rddr") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "kube-api-access-7rddr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:03.219435 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.219405 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:03.219515 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.219492 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:03.219556 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.219505 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:03.227465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.227439 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7484c312-3831-47c9-9628-8632592ea36c" (UID: "7484c312-3831-47c9-9628-8632592ea36c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317721 2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-installation-pull-secrets\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317754 2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7484c312-3831-47c9-9628-8632592ea36c-image-registry-private-configuration\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317789 2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-registry-certificates\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317819 2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-registry-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317833 2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-bound-sa-token\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.317847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317844 2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7484c312-3831-47c9-9628-8632592ea36c-trusted-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.318146 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317856 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rddr\" (UniqueName: \"kubernetes.io/projected/7484c312-3831-47c9-9628-8632592ea36c-kube-api-access-7rddr\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.318146 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.317868 2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7484c312-3831-47c9-9628-8632592ea36c-ca-trust-extracted\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 21:50:03.467292 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.467263 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:50:03.472170 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:03.472144 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85f4b4d747-m9xlq"] Jul 21 21:50:04.511960 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:04.511925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:04.595706 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:04.595674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:04.663759 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:04.663734 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7484c312-3831-47c9-9628-8632592ea36c" path="/var/lib/kubelet/pods/7484c312-3831-47c9-9628-8632592ea36c/volumes" Jul 21 21:50:04.699537 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:04.699513 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:04.752318 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:04.752284 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:05.001710 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:05.001675 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:05.975251 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:05.975216 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:09.130229 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:09.130196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:09.187474 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:09.187440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:10.136682 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:10.136648 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:10.654676 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:10.654591 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:10.756564 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:10.756527 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:11.263003 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:11.262956 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:11.365947 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:11.365909 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:18.013776 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:18.013745 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:18.117961 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:18.117925 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:21.115527 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:21.115479 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:21.216921 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:21.216885 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:21.321799 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:21.321768 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:24.112241 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:24.112213 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gzkz" Jul 21 21:50:25.591942 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:25.591904 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:25.692862 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:25.692832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:25.794172 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:25.794145 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:33.137922 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:33.137882 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:50:33.196821 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:50:33.196774 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:15.855445 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:15.855410 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:15.920680 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:15.920646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:24.486661 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:24.486596 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:27.744016 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:27.743978 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:27.804995 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:27.804966 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:29.708086 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:29.708051 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:29.762802 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:29.762775 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:33.024849 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:33.024794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:33.125699 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:33.125667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:33.180660 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:33.180630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:40.441823 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:40.441787 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:51:40.543033 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:51:40.542986 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:06.091179 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:06.091146 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:06.194180 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:06.194149 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:25.515306 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:25.515273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:26.193984 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:26.193954 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:31.460396 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:31.460359 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:31.560278 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:31.560251 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:57.888061 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:57.888027 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:52:59.224819 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:52:59.224782 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:01.491591 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:01.491501 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:01.552945 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:01.552909 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:02.987473 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:02.987440 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:03.088721 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:03.088690 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:03.189172 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:03.189139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:03.314940 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:03.314859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:03.388782 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:03.388747 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:05.611532 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:05.611494 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:06.016830 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:06.016753 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:08.296788 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:08.296754 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:08.397162 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:08.397131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:12.103368 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:12.103330 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:12.205351 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:12.205278 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:13.821184 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:13.821147 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:19.783733 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:19.783696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:19.885956 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:19.885927 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:20.986457 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:20.986417 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:21.044688 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:21.044659 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:22.603738 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:22.603709 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:31.337662 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:31.337630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:31.391996 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:31.391969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:37.493869 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:37.493835 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:38.506811 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.506766 2555 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 21:53:38.731001 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.730870 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64"] Jul 21 21:53:38.733250 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.731114 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7484c312-3831-47c9-9628-8632592ea36c" containerName="registry" Jul 21 21:53:38.733250 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.731126 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7484c312-3831-47c9-9628-8632592ea36c" containerName="registry" Jul 21 21:53:38.733250 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.731167 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7484c312-3831-47c9-9628-8632592ea36c" containerName="registry" Jul 21 21:53:38.733250 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.733219 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.737718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.737697 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 21:53:38.738221 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.738188 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 21:53:38.738221 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.738200 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 21:53:38.738411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.738275 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 21:53:38.738411 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.738401 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdpkb\"" Jul 21 21:53:38.750583 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.750561 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:38.758641 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.758573 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64"] Jul 21 21:53:38.849380 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.849348 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.849380 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.849382 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.849582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.849403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.849582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.849489 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw256\" (UniqueName: \"kubernetes.io/projected/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-kube-api-access-sw256\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.849582 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.849558 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.939306 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.939275 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:38.950344 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950321 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.950414 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950359 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.950414 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950382 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.950414 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950405 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.950574 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950435 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw256\" (UniqueName: \"kubernetes.io/projected/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-kube-api-access-sw256\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.950872 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.950849 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.952592 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.952566 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.952808 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.952792 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.952859 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.952839 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:38.960557 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:38.960528 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw256\" (UniqueName: \"kubernetes.io/projected/f3b6e149-0a77-4fd9-8a81-ca2e4763c237-kube-api-access-sw256\") pod \"opendatahub-operator-controller-manager-54667d85dc-dgq64\" (UID: \"f3b6e149-0a77-4fd9-8a81-ca2e4763c237\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:39.033271 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.033180 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:39.042616 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.042579 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:39.132888 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.132852 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:39.179936 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.179914 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64"] Jul 21 21:53:39.182476 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:53:39.182453 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b6e149_0a77_4fd9_8a81_ca2e4763c237.slice/crio-cdd27732ae18c365699cfd983519914b157af8f760a647335b9819f88477748a WatchSource:0}: Error finding container cdd27732ae18c365699cfd983519914b157af8f760a647335b9819f88477748a: Status 404 returned error can't find the container with id cdd27732ae18c365699cfd983519914b157af8f760a647335b9819f88477748a Jul 21 21:53:39.198230 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.198207 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:39.266673 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.266631 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:39.867339 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:39.867302 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" event={"ID":"f3b6e149-0a77-4fd9-8a81-ca2e4763c237","Type":"ContainerStarted","Data":"cdd27732ae18c365699cfd983519914b157af8f760a647335b9819f88477748a"} Jul 21 21:53:40.933898 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:40.933857 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:41.044392 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:41.044360 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:42.879718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:42.879682 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" event={"ID":"f3b6e149-0a77-4fd9-8a81-ca2e4763c237","Type":"ContainerStarted","Data":"6980c2d8ec7016fd90ae53ffcbbece35331169f46c5819547597218577711583"} Jul 21 21:53:43.884457 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:43.884421 2555 generic.go:356] "Generic (PLEG): container finished" podID="f3b6e149-0a77-4fd9-8a81-ca2e4763c237" containerID="6980c2d8ec7016fd90ae53ffcbbece35331169f46c5819547597218577711583" exitCode=0 Jul 21 21:53:43.884866 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:43.884481 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" event={"ID":"f3b6e149-0a77-4fd9-8a81-ca2e4763c237","Type":"ContainerDied","Data":"6980c2d8ec7016fd90ae53ffcbbece35331169f46c5819547597218577711583"} Jul 21 21:53:43.885016 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:43.885001 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:53:44.889514 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:44.889482 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" event={"ID":"f3b6e149-0a77-4fd9-8a81-ca2e4763c237","Type":"ContainerStarted","Data":"d110ad070ff19a04751e15e3f6d41018faa229e17f83e1eede199dc98dcfc4e3"} Jul 21 21:53:44.889969 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:44.889639 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:44.911720 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:44.911670 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" podStartSLOduration=2.46364924 podStartE2EDuration="6.911653068s" totalImagesPullingTime="3.423341104s" totalInitContainerRuntime="1.024662724s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:44.911396497 +0000 UTC m=+306.762156328" watchObservedRunningTime="2026-07-21 21:53:44.911653068 +0000 UTC m=+306.762412900" Jul 21 21:53:50.508405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:50.508371 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:50.608851 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:50.608818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:55.894720 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:55.894688 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-dgq64" Jul 21 21:53:57.973679 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:57.973630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:58.077056 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:58.077017 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:58.837136 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:58.837110 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:58.960847 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:58.960813 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:59.083293 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.083259 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:59.168051 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.167965 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:59.554478 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.554395 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:59.692258 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.692225 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-68vxd"] Jul 21 21:53:59.695217 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.695201 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.697708 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.697685 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 21:53:59.697835 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.697767 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 21:53:59.697903 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.697852 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k4z8d\"" Jul 21 21:53:59.698834 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.698815 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 21:53:59.698945 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.698815 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 21:53:59.705117 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.705093 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-68vxd"] Jul 21 21:53:59.798497 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.798455 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:53:59.817139 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.817062 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d01a020-ddca-4912-9446-3dd9e3691913-tmp\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.817139 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.817103 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.817313 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.817178 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grbb\" (UniqueName: \"kubernetes.io/projected/5d01a020-ddca-4912-9446-3dd9e3691913-kube-api-access-4grbb\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.918317 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.918274 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d01a020-ddca-4912-9446-3dd9e3691913-tmp\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.918509 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.918336 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.918509 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.918395 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grbb\" (UniqueName: \"kubernetes.io/projected/5d01a020-ddca-4912-9446-3dd9e3691913-kube-api-access-4grbb\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.918509 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:53:59.918444 2555 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 21:53:59.918652 ip-10-0-130-224 kubenswrapper[2555]: E0721 21:53:59.918530 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs podName:5d01a020-ddca-4912-9446-3dd9e3691913 nodeName:}" failed. No retries permitted until 2026-07-21 21:54:00.41850976 +0000 UTC m=+322.269269586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs") pod "kube-auth-proxy-594787b58-68vxd" (UID: "5d01a020-ddca-4912-9446-3dd9e3691913") : secret "kube-auth-proxy-tls" not found Jul 21 21:53:59.920680 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.920647 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d01a020-ddca-4912-9446-3dd9e3691913-tmp\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.929044 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.929017 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grbb\" (UniqueName: \"kubernetes.io/projected/5d01a020-ddca-4912-9446-3dd9e3691913-kube-api-access-4grbb\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:53:59.998724 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:53:59.998690 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:00.143730 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.143698 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:00.421960 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.421876 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:54:00.424310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.424289 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d01a020-ddca-4912-9446-3dd9e3691913-tls-certs\") pod \"kube-auth-proxy-594787b58-68vxd\" (UID: \"5d01a020-ddca-4912-9446-3dd9e3691913\") " pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:54:00.605365 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.605329 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" Jul 21 21:54:00.715766 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.715734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:00.740918 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.740874 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-68vxd"] Jul 21 21:54:00.743545 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:54:00.743520 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d01a020_ddca_4912_9446_3dd9e3691913.slice/crio-3ccb3fa47eda3abf39298db60303167771bd263d87b78de78c0f7b8ea398c83a WatchSource:0}: Error finding container 3ccb3fa47eda3abf39298db60303167771bd263d87b78de78c0f7b8ea398c83a: Status 404 returned error can't find the container with id 3ccb3fa47eda3abf39298db60303167771bd263d87b78de78c0f7b8ea398c83a Jul 21 21:54:00.802583 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.802552 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:00.945394 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:00.945313 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" event={"ID":"5d01a020-ddca-4912-9446-3dd9e3691913","Type":"ContainerStarted","Data":"3ccb3fa47eda3abf39298db60303167771bd263d87b78de78c0f7b8ea398c83a"} Jul 21 21:54:01.715306 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:01.715273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:03.451858 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:03.451820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:03.504575 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:03.504538 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:03.958755 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:03.958724 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" event={"ID":"5d01a020-ddca-4912-9446-3dd9e3691913","Type":"ContainerStarted","Data":"0398ae3c62c8e51d92bfc871dcd1fc28b3934a4926217b7af640d740fcb8c725"} Jul 21 21:54:03.976856 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:03.976754 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594787b58-68vxd" podStartSLOduration=2.036980127 podStartE2EDuration="4.976735727s" totalImagesPullingTime="2.9397556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:03.975768652 +0000 UTC m=+325.826528483" watchObservedRunningTime="2026-07-21 21:54:03.976735727 +0000 UTC m=+325.827495558" Jul 21 21:54:05.411234 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:05.411196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:05.511774 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:05.511742 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:07.054564 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:07.054534 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:07.184441 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:07.184410 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:07.776328 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:07.776292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:07.945823 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:07.945792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:08.047519 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.047439 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:08.242429 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.242395 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk"] Jul 21 21:54:08.245873 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.245850 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.249471 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.249451 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 21:54:08.250310 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.250295 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 21:54:08.250503 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.250489 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 21:54:08.250578 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.250506 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njsh6\"" Jul 21 21:54:08.251013 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.250994 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 21:54:08.276257 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.276230 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk"] Jul 21 21:54:08.386435 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386397 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366c5814-8054-4caf-bab0-c8633d3959e2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386447 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386521 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386553 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386640 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386593 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66q5\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-kube-api-access-j66q5\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386791 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386651 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.386791 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.386688 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487760 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487721 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487930 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487777 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366c5814-8054-4caf-bab0-c8633d3959e2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487930 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487808 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487930 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487845 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487930 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487873 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.487930 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487907 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66q5\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-kube-api-access-j66q5\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.488178 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.487932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.488514 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.488490 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.490342 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.490317 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/366c5814-8054-4caf-bab0-c8633d3959e2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.490453 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.490394 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.490704 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.490678 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.490777 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.490686 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/366c5814-8054-4caf-bab0-c8633d3959e2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.497406 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.497380 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66q5\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-kube-api-access-j66q5\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.498621 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.498588 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366c5814-8054-4caf-bab0-c8633d3959e2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4n2wk\" (UID: \"366c5814-8054-4caf-bab0-c8633d3959e2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.555162 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.555126 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:08.700956 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.700790 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk"] Jul 21 21:54:08.703457 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:54:08.703427 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366c5814_8054_4caf_bab0_c8633d3959e2.slice/crio-f5ef52c915b6a11c9d77a3b7c756b9ae998d33f6eb210974e8f15e32dc9cd774 WatchSource:0}: Error finding container f5ef52c915b6a11c9d77a3b7c756b9ae998d33f6eb210974e8f15e32dc9cd774: Status 404 returned error can't find the container with id f5ef52c915b6a11c9d77a3b7c756b9ae998d33f6eb210974e8f15e32dc9cd774 Jul 21 21:54:08.911508 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.911433 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:08.977205 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:08.977171 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" event={"ID":"366c5814-8054-4caf-bab0-c8633d3959e2","Type":"ContainerStarted","Data":"f5ef52c915b6a11c9d77a3b7c756b9ae998d33f6eb210974e8f15e32dc9cd774"} Jul 21 21:54:11.110272 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.110221 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 21:54:11.110631 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.110310 2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 21 21:54:11.274572 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.274545 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.368390 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.368304 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.441590 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.441560 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw"] Jul 21 21:54:11.444919 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.444899 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.448624 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.448590 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-chqxj\"" Jul 21 21:54:11.448993 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.448979 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 21:54:11.457214 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.457138 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.459335 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.459312 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw"] Jul 21 21:54:11.616810 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.616773 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslcj\" (UniqueName: \"kubernetes.io/projected/1af52da3-84bd-4c78-8b5f-ce023a1a2554-kube-api-access-vslcj\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.616810 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.616815 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af52da3-84bd-4c78-8b5f-ce023a1a2554-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.647142 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.647058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.717947 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.717911 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslcj\" (UniqueName: \"kubernetes.io/projected/1af52da3-84bd-4c78-8b5f-ce023a1a2554-kube-api-access-vslcj\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.718150 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.717961 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af52da3-84bd-4c78-8b5f-ce023a1a2554-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.721272 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.721238 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af52da3-84bd-4c78-8b5f-ce023a1a2554-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.729433 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.729401 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslcj\" (UniqueName: \"kubernetes.io/projected/1af52da3-84bd-4c78-8b5f-ce023a1a2554-kube-api-access-vslcj\") pod \"llmisvc-controller-manager-777c6bd7d5-f5tbw\" (UID: \"1af52da3-84bd-4c78-8b5f-ce023a1a2554\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.747246 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.747202 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.757235 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.757216 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:11.846596 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.846562 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:11.907093 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.907070 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw"] Jul 21 21:54:11.908961 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:54:11.908935 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1af52da3_84bd_4c78_8b5f_ce023a1a2554.slice/crio-78d1aeeac7c2d247a095bbaf3d41ef5dd3d40e79fc704f9f63ced8eafb8d7b56 WatchSource:0}: Error finding container 78d1aeeac7c2d247a095bbaf3d41ef5dd3d40e79fc704f9f63ced8eafb8d7b56: Status 404 returned error can't find the container with id 78d1aeeac7c2d247a095bbaf3d41ef5dd3d40e79fc704f9f63ced8eafb8d7b56 Jul 21 21:54:11.990467 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.990441 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" event={"ID":"366c5814-8054-4caf-bab0-c8633d3959e2","Type":"ContainerStarted","Data":"fd788578f96f418ea463ca986940b86211c3f76c65997e7fa0064e8224e50a1c"} Jul 21 21:54:11.990635 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.990508 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:11.991677 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:11.991649 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" event={"ID":"1af52da3-84bd-4c78-8b5f-ce023a1a2554","Type":"ContainerStarted","Data":"78d1aeeac7c2d247a095bbaf3d41ef5dd3d40e79fc704f9f63ced8eafb8d7b56"} Jul 21 21:54:12.015757 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:12.015720 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" podStartSLOduration=1.611459483 podStartE2EDuration="4.015707007s" totalImagesPullingTime="2.404247524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:12.014683554 +0000 UTC m=+333.865443385" watchObservedRunningTime="2026-07-21 21:54:12.015707007 +0000 UTC m=+333.866466839" Jul 21 21:54:13.000193 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:13.000159 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4n2wk" Jul 21 21:54:15.007405 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:15.007372 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" event={"ID":"1af52da3-84bd-4c78-8b5f-ce023a1a2554","Type":"ContainerStarted","Data":"337f0697a61c0657cf8ceff52347b07bae13c504dc5fed82c33d1aae8be7e969"} Jul 21 21:54:15.007850 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:15.007493 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:15.026882 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:15.026833 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" podStartSLOduration=1.462378157 podStartE2EDuration="4.026817346s" totalImagesPullingTime="2.564439189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:15.025516855 +0000 UTC m=+336.876276687" watchObservedRunningTime="2026-07-21 21:54:15.026817346 +0000 UTC m=+336.877577211" Jul 21 21:54:43.014527 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:43.014441 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:46.013896 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:46.013867 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f5tbw" Jul 21 21:54:59.411430 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:59.411385 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:59.529712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:59.529681 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:54:59.647747 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:54:59.647713 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:00.369769 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:00.369741 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:00.576000 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:00.575969 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:21.353908 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:21.353873 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:21.458167 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:21.458139 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:23.098148 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:23.098115 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:25.846161 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:25.846126 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:26.630199 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:26.630160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:29.357044 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.357002 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk"] Jul 21 21:55:29.362464 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.362433 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:29.365244 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.365223 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:55:29.366351 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.366333 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:55:29.366351 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.366343 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j7jrf\"" Jul 21 21:55:29.374495 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.374471 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk"] Jul 21 21:55:29.504356 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.504318 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn4m\" (UniqueName: \"kubernetes.io/projected/5092bd18-788a-40ae-a44a-96efb792426a-kube-api-access-6nn4m\") pod \"authorino-operator-75cc8d5cb7-hq8fk\" (UID: \"5092bd18-788a-40ae-a44a-96efb792426a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:29.605667 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.605629 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn4m\" (UniqueName: \"kubernetes.io/projected/5092bd18-788a-40ae-a44a-96efb792426a-kube-api-access-6nn4m\") pod \"authorino-operator-75cc8d5cb7-hq8fk\" (UID: \"5092bd18-788a-40ae-a44a-96efb792426a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:29.616773 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.616706 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn4m\" (UniqueName: \"kubernetes.io/projected/5092bd18-788a-40ae-a44a-96efb792426a-kube-api-access-6nn4m\") pod \"authorino-operator-75cc8d5cb7-hq8fk\" (UID: \"5092bd18-788a-40ae-a44a-96efb792426a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:29.672353 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.672323 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:29.805807 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:29.805776 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk"] Jul 21 21:55:29.808811 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:55:29.808786 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5092bd18_788a_40ae_a44a_96efb792426a.slice/crio-b9c05785de4051726a8a1239cc03c6fd136e2cc0536795b5f01ad9f866dd6b9c WatchSource:0}: Error finding container b9c05785de4051726a8a1239cc03c6fd136e2cc0536795b5f01ad9f866dd6b9c: Status 404 returned error can't find the container with id b9c05785de4051726a8a1239cc03c6fd136e2cc0536795b5f01ad9f866dd6b9c Jul 21 21:55:30.242800 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:30.242772 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:30.298722 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:30.298683 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" event={"ID":"5092bd18-788a-40ae-a44a-96efb792426a","Type":"ContainerStarted","Data":"b9c05785de4051726a8a1239cc03c6fd136e2cc0536795b5f01ad9f866dd6b9c"} Jul 21 21:55:30.557457 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:30.557372 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:30.662211 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:30.662178 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:31.957438 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:31.957400 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:32.064619 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.064576 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:32.170345 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.170310 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:32.309015 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.308984 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" event={"ID":"5092bd18-788a-40ae-a44a-96efb792426a","Type":"ContainerStarted","Data":"b4b3c168a865be90a04b470ee12a594c30fc9d6c5c2d8c3ea4760a4fdf49ef52"} Jul 21 21:55:32.309207 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.309101 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:32.327667 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.327594 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" podStartSLOduration=0.919436641 podStartE2EDuration="3.327575911s" totalImagesPullingTime="2.40813927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:32.326050074 +0000 UTC m=+414.176809907" watchObservedRunningTime="2026-07-21 21:55:32.327575911 +0000 UTC m=+414.178335742" Jul 21 21:55:32.901934 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:32.901897 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:35.450665 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:35.450630 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:37.487332 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.487294 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w"] Jul 21 21:55:37.490474 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.490458 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:37.495071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.495049 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 21:55:37.495208 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.495082 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fxxdr\"" Jul 21 21:55:37.504338 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.504311 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w"] Jul 21 21:55:37.574997 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.574958 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjccf\" (UniqueName: \"kubernetes.io/projected/2436d5fe-0ef7-41df-b8a6-ccd215217b93-kube-api-access-fjccf\") pod \"dns-operator-controller-manager-864dd78c4c-zpx2w\" (UID: \"2436d5fe-0ef7-41df-b8a6-ccd215217b93\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:37.675995 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.675951 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjccf\" (UniqueName: \"kubernetes.io/projected/2436d5fe-0ef7-41df-b8a6-ccd215217b93-kube-api-access-fjccf\") pod \"dns-operator-controller-manager-864dd78c4c-zpx2w\" (UID: \"2436d5fe-0ef7-41df-b8a6-ccd215217b93\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:37.682530 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.682500 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:37.684919 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.684896 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjccf\" (UniqueName: \"kubernetes.io/projected/2436d5fe-0ef7-41df-b8a6-ccd215217b93-kube-api-access-fjccf\") pod \"dns-operator-controller-manager-864dd78c4c-zpx2w\" (UID: \"2436d5fe-0ef7-41df-b8a6-ccd215217b93\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:37.800071 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.799995 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:37.941391 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:37.941367 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w"] Jul 21 21:55:37.944093 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:55:37.944069 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2436d5fe_0ef7_41df_b8a6_ccd215217b93.slice/crio-53d6e79f3186a84007929d06fdd68473562960896f86ffb1205f09fc0e72afe5 WatchSource:0}: Error finding container 53d6e79f3186a84007929d06fdd68473562960896f86ffb1205f09fc0e72afe5: Status 404 returned error can't find the container with id 53d6e79f3186a84007929d06fdd68473562960896f86ffb1205f09fc0e72afe5 Jul 21 21:55:38.336386 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:38.336351 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" event={"ID":"2436d5fe-0ef7-41df-b8a6-ccd215217b93","Type":"ContainerStarted","Data":"53d6e79f3186a84007929d06fdd68473562960896f86ffb1205f09fc0e72afe5"} Jul 21 21:55:39.925758 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:39.925734 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 21:55:40.271202 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:40.271121 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:40.346436 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:40.346401 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" event={"ID":"2436d5fe-0ef7-41df-b8a6-ccd215217b93","Type":"ContainerStarted","Data":"51bda90527598cef106d573919fb14a3317d60bf82ad147a6a13201e1cdeddc6"} Jul 21 21:55:40.346629 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:40.346532 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:40.363961 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:40.363905 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" podStartSLOduration=1.388300207 podStartE2EDuration="3.363888371s" totalImagesPullingTime="1.975588164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:40.362846954 +0000 UTC m=+422.213606786" watchObservedRunningTime="2026-07-21 21:55:40.363888371 +0000 UTC m=+422.214648201" Jul 21 21:55:42.834799 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:42.834751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:43.091152 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:43.091033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:43.315982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:43.315952 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hq8fk" Jul 21 21:55:44.641159 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:44.641114 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:44.705407 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:44.705366 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:45.043743 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:45.043627 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:45.101701 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:45.101654 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:48.440830 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:48.440779 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:48.542302 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:48.542243 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:51.353397 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:51.353368 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zpx2w" Jul 21 21:55:53.570409 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.570379 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw"] Jul 21 21:55:53.573747 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.573724 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:53.576272 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.576247 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-44pmh\"" Jul 21 21:55:53.582792 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.582767 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw"] Jul 21 21:55:53.609223 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.609189 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbnx\" (UniqueName: \"kubernetes.io/projected/f5398776-cd1b-4e44-8791-98ea74723b22-kube-api-access-gvbnx\") pod \"limitador-operator-controller-manager-859d7b78bd-tlkmw\" (UID: \"f5398776-cd1b-4e44-8791-98ea74723b22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:53.709814 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.709774 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbnx\" (UniqueName: \"kubernetes.io/projected/f5398776-cd1b-4e44-8791-98ea74723b22-kube-api-access-gvbnx\") pod \"limitador-operator-controller-manager-859d7b78bd-tlkmw\" (UID: \"f5398776-cd1b-4e44-8791-98ea74723b22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:53.723047 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.723009 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbnx\" (UniqueName: \"kubernetes.io/projected/f5398776-cd1b-4e44-8791-98ea74723b22-kube-api-access-gvbnx\") pod \"limitador-operator-controller-manager-859d7b78bd-tlkmw\" (UID: \"f5398776-cd1b-4e44-8791-98ea74723b22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:53.885526 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:53.885430 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:54.018295 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:54.018272 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw"] Jul 21 21:55:54.020999 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:55:54.020973 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5398776_cd1b_4e44_8791_98ea74723b22.slice/crio-86d0e51325755c719c5d9b89d599a93966b4ff254912c2c324373d3458b874b3 WatchSource:0}: Error finding container 86d0e51325755c719c5d9b89d599a93966b4ff254912c2c324373d3458b874b3: Status 404 returned error can't find the container with id 86d0e51325755c719c5d9b89d599a93966b4ff254912c2c324373d3458b874b3 Jul 21 21:55:54.404852 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:54.404816 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" event={"ID":"f5398776-cd1b-4e44-8791-98ea74723b22","Type":"ContainerStarted","Data":"86d0e51325755c719c5d9b89d599a93966b4ff254912c2c324373d3458b874b3"} Jul 21 21:55:54.515059 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:54.515026 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:54.613676 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:54.613646 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:56.417500 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:56.417465 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" event={"ID":"f5398776-cd1b-4e44-8791-98ea74723b22","Type":"ContainerStarted","Data":"88646480cfdb2b653d9345defe0f75c3d5931b9a2853878cbb478bbb085149de"} Jul 21 21:55:56.417974 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:56.417532 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:55:56.436496 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:56.436447 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" podStartSLOduration=1.88557074 podStartE2EDuration="3.43643204s" totalImagesPullingTime="1.5508613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:55:56.435284422 +0000 UTC m=+438.286044255" watchObservedRunningTime="2026-07-21 21:55:56.43643204 +0000 UTC m=+438.287191871" Jul 21 21:55:58.781746 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:58.781711 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:55:58.881115 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:55:58.881079 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:02.264245 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:02.264165 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:02.365288 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:02.365259 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:05.793957 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:05.793918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:05.894024 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:05.893994 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:07.423439 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:07.423413 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlkmw" Jul 21 21:56:12.276703 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:12.276660 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:12.329933 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:12.329907 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:16.641100 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:16.641063 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:16.741264 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:16.741234 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:22.077905 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:22.077871 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:22.179230 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:22.179198 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:22.410534 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:22.410449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:22.507781 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:22.507751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:22.607982 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:22.607948 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:26.038110 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:26.038077 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:26.137969 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:26.137935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:29.613891 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:29.613854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:31.892708 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:31.892678 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:31.993157 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:31.993102 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:37.263085 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:37.263052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:37.360522 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:37.360486 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:37.420784 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:37.420751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:38.093973 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:38.093935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:38.283927 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:38.283894 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:55.586232 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:55.586198 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:55.687985 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:55.687953 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:56:56.079820 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:56:56.079790 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:04.530195 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:04.530160 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:04.632764 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:04.632732 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:05.987026 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:05.986994 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:08.613683 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:08.613647 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:08.894833 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:08.894750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:15.080851 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:15.080820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:15.177923 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:15.177887 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:15.231979 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:15.231947 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:18.418067 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:18.418031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:18.516287 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:18.516256 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:34.624536 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:34.624443 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:34.726743 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:34.726716 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:37.327180 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:37.327143 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:37.428557 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:37.428525 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:50.074482 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.074444 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:50.169960 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.169929 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:50.659590 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.659563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:50.709623 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.709577 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55794f648-fs75v"] Jul 21 21:57:50.712970 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.712947 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.715717 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.715695 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:57:50.715819 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.715695 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 21:57:50.715989 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.715975 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 21:57:50.723701 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.723670 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55794f648-fs75v"] Jul 21 21:57:50.851495 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.851464 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxnj\" (UniqueName: \"kubernetes.io/projected/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-kube-api-access-8wxnj\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.851495 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.851503 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-openshift-service-ca\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.851718 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.851622 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-maas-api-tls\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.952138 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.952016 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-maas-api-tls\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.952138 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.952107 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxnj\" (UniqueName: \"kubernetes.io/projected/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-kube-api-access-8wxnj\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.952370 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.952137 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-openshift-service-ca\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.952993 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.952971 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-openshift-service-ca\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.954660 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.954639 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-maas-api-tls\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:50.963878 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:50.963855 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxnj\" (UniqueName: \"kubernetes.io/projected/18b451fa-e9cf-44dc-a8a6-62fdc99f5984-kube-api-access-8wxnj\") pod \"maas-api-55794f648-fs75v\" (UID: \"18b451fa-e9cf-44dc-a8a6-62fdc99f5984\") " pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:51.023737 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.023704 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:51.096431 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.096396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:51.169748 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.169726 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55794f648-fs75v"] Jul 21 21:57:51.171894 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:57:51.171859 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b451fa_e9cf_44dc_a8a6_62fdc99f5984.slice/crio-09fcf5b024813f799f1cdc0bd95669fa72ce12936fed31258f83c893c38cfafc WatchSource:0}: Error finding container 09fcf5b024813f799f1cdc0bd95669fa72ce12936fed31258f83c893c38cfafc: Status 404 returned error can't find the container with id 09fcf5b024813f799f1cdc0bd95669fa72ce12936fed31258f83c893c38cfafc Jul 21 21:57:51.259578 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.259459 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:51.357661 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.357592 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:57:51.883190 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:51.883144 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" event={"ID":"18b451fa-e9cf-44dc-a8a6-62fdc99f5984","Type":"ContainerStarted","Data":"09fcf5b024813f799f1cdc0bd95669fa72ce12936fed31258f83c893c38cfafc"} Jul 21 21:57:53.893521 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:53.893486 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" event={"ID":"18b451fa-e9cf-44dc-a8a6-62fdc99f5984","Type":"ContainerStarted","Data":"810db6d165c32ea3b61691ee438ea143e26295ca98506338c7c68eb1bb429f09"} Jul 21 21:57:53.893985 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:53.893554 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:57:53.910712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:53.910642 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" podStartSLOduration=1.540755153 podStartE2EDuration="3.910624468s" totalImagesPullingTime="2.369869315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:57:53.910324361 +0000 UTC m=+555.761084193" watchObservedRunningTime="2026-07-21 21:57:53.910624468 +0000 UTC m=+555.761384302" Jul 21 21:57:59.904015 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:57:59.903982 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55794f648-fs75v" Jul 21 21:58:01.299767 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:01.299736 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:01.403983 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:01.403958 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:04.934170 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:04.934124 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:04.994504 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:04.994474 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:14.490856 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:14.490820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:38.484854 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.484811 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg"] Jul 21 21:58:38.488595 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.488575 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.492892 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.492860 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 21:58:38.493023 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.492941 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 21:58:38.493023 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.492999 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 21:58:38.493131 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.493067 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kccl4\"" Jul 21 21:58:38.498125 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.498100 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg"] Jul 21 21:58:38.517799 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.517765 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:38.647520 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647486 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntxn\" (UniqueName: \"kubernetes.io/projected/92640ae1-df20-42bc-9180-89a937874eb4-kube-api-access-jntxn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.647712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647529 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.647712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.647712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647667 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.647712 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647695 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92640ae1-df20-42bc-9180-89a937874eb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.647880 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.647720 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.748919 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.748963 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92640ae1-df20-42bc-9180-89a937874eb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.748985 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749021 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntxn\" (UniqueName: \"kubernetes.io/projected/92640ae1-df20-42bc-9180-89a937874eb4-kube-api-access-jntxn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749054 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.749097 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749077 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.755300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749399 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.755300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749437 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.755300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.749500 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.755300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.751215 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92640ae1-df20-42bc-9180-89a937874eb4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.755300 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.751471 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92640ae1-df20-42bc-9180-89a937874eb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.761679 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.761651 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntxn\" (UniqueName: \"kubernetes.io/projected/92640ae1-df20-42bc-9180-89a937874eb4-kube-api-access-jntxn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f8cfg\" (UID: \"92640ae1-df20-42bc-9180-89a937874eb4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.799601 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.799566 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:38.934753 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:38.934730 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg"] Jul 21 21:58:38.936923 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:58:38.936892 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92640ae1_df20_42bc_9180_89a937874eb4.slice/crio-0e24bcd1ba09160b39edd3b7be4dc4703522768fb7d57fd8f5b8c094a6609455 WatchSource:0}: Error finding container 0e24bcd1ba09160b39edd3b7be4dc4703522768fb7d57fd8f5b8c094a6609455: Status 404 returned error can't find the container with id 0e24bcd1ba09160b39edd3b7be4dc4703522768fb7d57fd8f5b8c094a6609455 Jul 21 21:58:39.077430 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:39.077397 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" event={"ID":"92640ae1-df20-42bc-9180-89a937874eb4","Type":"ContainerStarted","Data":"0e24bcd1ba09160b39edd3b7be4dc4703522768fb7d57fd8f5b8c094a6609455"} Jul 21 21:58:40.084676 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:40.084638 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" event={"ID":"92640ae1-df20-42bc-9180-89a937874eb4","Type":"ContainerStarted","Data":"7f899813bd27c9a507684bf420e3b943ea957e6ad50ff94b8db943cf508c5bc0"} Jul 21 21:58:40.105112 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:40.105057 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" podStartSLOduration=1.045777948 podStartE2EDuration="2.105041166s" totalImagesPullingTime="1.059263218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:58:40.103586933 +0000 UTC m=+601.954346765" watchObservedRunningTime="2026-07-21 21:58:40.105041166 +0000 UTC m=+601.955800997" Jul 21 21:58:45.088453 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.088416 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr"] Jul 21 21:58:45.091880 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.091860 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.094522 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.094502 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 21:58:45.109379 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.109347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr"] Jul 21 21:58:45.126541 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.126511 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:45.201216 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201177 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.201216 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201223 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt4p\" (UniqueName: \"kubernetes.io/projected/66c52393-890f-4727-952d-501b3a1ad3f3-kube-api-access-flt4p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.201465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201249 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.201465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201357 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c52393-890f-4727-952d-501b3a1ad3f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.201465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201403 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.201465 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.201444 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302307 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302269 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302307 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt4p\" (UniqueName: \"kubernetes.io/projected/66c52393-890f-4727-952d-501b3a1ad3f3-kube-api-access-flt4p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302526 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302334 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302526 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302363 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c52393-890f-4727-952d-501b3a1ad3f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302526 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302383 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302526 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302411 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302834 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302810 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302943 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302858 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.302943 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.302927 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.304679 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.304659 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c52393-890f-4727-952d-501b3a1ad3f3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.304963 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.304942 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c52393-890f-4727-952d-501b3a1ad3f3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.312281 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.312260 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt4p\" (UniqueName: \"kubernetes.io/projected/66c52393-890f-4727-952d-501b3a1ad3f3-kube-api-access-flt4p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cgmdr\" (UID: \"66c52393-890f-4727-952d-501b3a1ad3f3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.403391 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.403294 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:45.543122 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.542965 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr"] Jul 21 21:58:45.545492 ip-10-0-130-224 kubenswrapper[2555]: W0721 21:58:45.545467 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c52393_890f_4727_952d_501b3a1ad3f3.slice/crio-01876970ef4cd8d9db027832294d112d52c15e9048b789cd04c582fbf9c7796d WatchSource:0}: Error finding container 01876970ef4cd8d9db027832294d112d52c15e9048b789cd04c582fbf9c7796d: Status 404 returned error can't find the container with id 01876970ef4cd8d9db027832294d112d52c15e9048b789cd04c582fbf9c7796d Jul 21 21:58:45.546994 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:45.546976 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:58:46.114822 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:46.114788 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" event={"ID":"66c52393-890f-4727-952d-501b3a1ad3f3","Type":"ContainerStarted","Data":"9072ce7652f5c6112833b27405cfa03b12d1478374b35f2361738c7ad301d83d"} Jul 21 21:58:46.115234 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:46.114838 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" event={"ID":"66c52393-890f-4727-952d-501b3a1ad3f3","Type":"ContainerStarted","Data":"01876970ef4cd8d9db027832294d112d52c15e9048b789cd04c582fbf9c7796d"} Jul 21 21:58:46.137842 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:46.137786 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" podStartSLOduration=0.926431136 podStartE2EDuration="1.137767006s" totalImagesPullingTime="211.33587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:58:46.136814989 +0000 UTC m=+607.987574821" watchObservedRunningTime="2026-07-21 21:58:46.137767006 +0000 UTC m=+607.988526836" Jul 21 21:58:48.799977 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:48.799938 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:48.800395 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:48.800014 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:48.812334 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:48.812303 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:49.139689 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:49.139656 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f8cfg" Jul 21 21:58:51.516206 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:51.516165 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:58:55.404038 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:55.403998 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:55.404453 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:55.404063 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:55.416829 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:55.416804 2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:58:56.167578 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:58:56.167551 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cgmdr" Jul 21 21:59:00.075412 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:59:00.075340 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:59:09.784089 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:59:09.784056 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 21:59:16.678189 ip-10-0-130-224 kubenswrapper[2555]: I0721 21:59:16.678153 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:00:00.154462 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.154423 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8"] Jul 21 22:00:00.157282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.157265 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" Jul 21 22:00:00.161002 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.160981 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:00:00.165429 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.165404 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8"] Jul 21 22:00:00.168391 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.168369 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" Jul 21 22:00:00.302362 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.302336 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8"] Jul 21 22:00:00.304386 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:00:00.304360 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30fdc68_8adf_4ad9_b378_b218fdd24b7f.slice/crio-07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f WatchSource:0}: Error finding container 07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f: Status 404 returned error can't find the container with id 07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f Jul 21 22:00:00.427206 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:00.427124 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" event={"ID":"e30fdc68-8adf-4ad9-b378-b218fdd24b7f","Type":"ContainerStarted","Data":"07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f"} Jul 21 22:00:05.451331 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:05.451237 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" event={"ID":"e30fdc68-8adf-4ad9-b378-b218fdd24b7f","Type":"ContainerStarted","Data":"4ee4df2995bd477639d087788355453a314d4ece585cd348191560a416fe69e2"} Jul 21 22:00:05.469339 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:05.469282 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" podStartSLOduration=0.700421581 podStartE2EDuration="5.469267061s" totalImagesPullingTime="4.76884548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:00:05.467762983 +0000 UTC m=+687.318522815" watchObservedRunningTime="2026-07-21 22:00:05.469267061 +0000 UTC m=+687.320026892" Jul 21 22:00:06.456501 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:06.456468 2555 generic.go:356] "Generic (PLEG): container finished" podID="e30fdc68-8adf-4ad9-b378-b218fdd24b7f" containerID="4ee4df2995bd477639d087788355453a314d4ece585cd348191560a416fe69e2" exitCode=0 Jul 21 22:00:06.456911 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:06.456527 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" event={"ID":"e30fdc68-8adf-4ad9-b378-b218fdd24b7f","Type":"ContainerDied","Data":"4ee4df2995bd477639d087788355453a314d4ece585cd348191560a416fe69e2"} Jul 21 22:00:07.583583 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:07.583560 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" Jul 21 22:00:08.466303 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:08.466276 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" Jul 21 22:00:08.466303 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:08.466300 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744520-4kzp8" event={"ID":"e30fdc68-8adf-4ad9-b378-b218fdd24b7f","Type":"ContainerDied","Data":"07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f"} Jul 21 22:00:08.466505 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:08.466349 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07f7b3f32fea3d56a1244db6a34642d7dbf8621d7746c24b614582f24374808f" Jul 21 22:00:25.066160 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:25.066121 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:00:45.765434 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:45.765344 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:00:45.861016 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:45.860987 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:00:47.024388 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:47.024347 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:00:47.125862 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:00:47.125829 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:01:53.481464 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:01:53.481427 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:02:53.773019 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:02:53.772991 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:02:55.311702 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:02:55.311672 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:35.256749 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:35.256699 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:35.403650 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:35.403590 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:35.588856 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:35.588812 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:42.415889 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:42.415855 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:42.511799 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:42.511733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:42.822899 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:42.822875 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:42.913863 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:42.913818 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:43.492693 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:43.492667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:50.281716 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:50.281685 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:03:54.642618 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:03:54.642586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:05:10.112101 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:05:10.112000 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:06:35.518373 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:06:35.518254 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:06:35.578911 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:06:35.578880 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:06:39.905508 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:06:39.905467 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:07:27.490099 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:07:27.490047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:07:27.591314 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:07:27.591282 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:11:36.737788 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:11:36.737752 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:11:49.792720 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:11:49.792689 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:12:12.031493 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:12:12.031463 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:12:13.520644 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:12:13.520585 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:14:30.211499 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:14:30.211466 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:14:40.690233 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:14:40.690196 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:00.202098 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.202061 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx"] Jul 21 22:15:00.202514 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.202427 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e30fdc68-8adf-4ad9-b378-b218fdd24b7f" containerName="cleanup" Jul 21 22:15:00.202514 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.202439 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e30fdc68-8adf-4ad9-b378-b218fdd24b7f" containerName="cleanup" Jul 21 22:15:00.202514 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.202497 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e30fdc68-8adf-4ad9-b378-b218fdd24b7f" containerName="cleanup" Jul 21 22:15:00.205429 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.205408 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" Jul 21 22:15:00.213541 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.213521 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:15:00.214945 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.214927 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" Jul 21 22:15:00.298597 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.298562 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx"] Jul 21 22:15:00.545135 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:00.545105 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx"] Jul 21 22:15:00.547523 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:15:00.547490 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5182e35_f604_421d_ab6a_4caccde0e368.slice/crio-3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654 WatchSource:0}: Error finding container 3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654: Status 404 returned error can't find the container with id 3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654 Jul 21 22:15:01.239502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:01.239467 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" event={"ID":"b5182e35-f604-421d-ab6a-4caccde0e368","Type":"ContainerStarted","Data":"195cfe12fcafe871f5bd0279b5ce126c96a00ef8dcf23934bfccf7ca3ae1e5c6"} Jul 21 22:15:01.239920 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:01.239517 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" event={"ID":"b5182e35-f604-421d-ab6a-4caccde0e368","Type":"ContainerStarted","Data":"3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654"} Jul 21 22:15:01.256857 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:01.256812 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" podStartSLOduration=1.256796263 podStartE2EDuration="1.256796263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:15:01.25609639 +0000 UTC m=+1583.106856222" watchObservedRunningTime="2026-07-21 22:15:01.256796263 +0000 UTC m=+1583.107556093" Jul 21 22:15:02.245129 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:02.245095 2555 generic.go:356] "Generic (PLEG): container finished" podID="b5182e35-f604-421d-ab6a-4caccde0e368" containerID="195cfe12fcafe871f5bd0279b5ce126c96a00ef8dcf23934bfccf7ca3ae1e5c6" exitCode=0 Jul 21 22:15:02.245548 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:02.245156 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" event={"ID":"b5182e35-f604-421d-ab6a-4caccde0e368","Type":"ContainerDied","Data":"195cfe12fcafe871f5bd0279b5ce126c96a00ef8dcf23934bfccf7ca3ae1e5c6"} Jul 21 22:15:03.373979 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:03.373957 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" Jul 21 22:15:04.254713 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:04.254687 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" Jul 21 22:15:04.254883 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:04.254735 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744535-nwbsx" event={"ID":"b5182e35-f604-421d-ab6a-4caccde0e368","Type":"ContainerDied","Data":"3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654"} Jul 21 22:15:04.254883 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:04.254783 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c591387914f7dbc7b81f561ad3f70b55390bd8304faecf8b44047ae8ac32654" Jul 21 22:15:22.305044 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:22.304983 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:43.732341 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:43.732251 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:46.743052 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:46.743010 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:46.834390 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:46.834357 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:47.238225 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:47.238191 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:53.115065 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:53.115028 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:53.215168 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:53.215140 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:53.351938 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:53.351909 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:53.665746 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:53.665716 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:53.764332 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:53.764287 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:55.660046 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.660012 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:55.741138 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.741098 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:55.742163 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.742141 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:15:55.742450 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.742439 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5182e35-f604-421d-ab6a-4caccde0e368" containerName="cleanup" Jul 21 22:15:55.742495 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.742452 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5182e35-f604-421d-ab6a-4caccde0e368" containerName="cleanup" Jul 21 22:15:55.742574 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.742505 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5182e35-f604-421d-ab6a-4caccde0e368" containerName="cleanup" Jul 21 22:15:55.745365 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.745349 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.748205 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.748186 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-678339c2-serving-cert\"" Jul 21 22:15:55.748319 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.748187 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:15:55.755179 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.755150 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:15:55.803847 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.803820 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:55.804229 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.804205 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.804372 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.804338 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw7l\" (UniqueName: \"kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.804424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.804416 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.905156 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.905131 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.905302 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.905161 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw7l\" (UniqueName: \"kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.905302 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.905189 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.905789 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.905767 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.907523 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.907501 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.916595 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.916543 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw7l\" (UniqueName: \"kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l\") pod \"maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:55.957362 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:55.957329 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:56.057897 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.057879 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:56.189280 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.189257 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:15:56.191262 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:15:56.191237 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04a085f_7038_4b59_b05b_289220dec8d7.slice/crio-7b5a76571028cd69d8556886db9567b262c5502417c533883190a55b0d37f2d7 WatchSource:0}: Error finding container 7b5a76571028cd69d8556886db9567b262c5502417c533883190a55b0d37f2d7: Status 404 returned error can't find the container with id 7b5a76571028cd69d8556886db9567b262c5502417c533883190a55b0d37f2d7 Jul 21 22:15:56.192647 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.192630 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 22:15:56.242764 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.242735 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:56.342235 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.342210 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:15:56.483450 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:56.483384 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" event={"ID":"a04a085f-7038-4b59-b05b-289220dec8d7","Type":"ContainerStarted","Data":"7b5a76571028cd69d8556886db9567b262c5502417c533883190a55b0d37f2d7"} Jul 21 22:15:57.489812 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:57.489773 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" event={"ID":"a04a085f-7038-4b59-b05b-289220dec8d7","Type":"ContainerStarted","Data":"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0"} Jul 21 22:15:57.490303 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:57.489845 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:15:57.513234 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:15:57.513183 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" podStartSLOduration=1.911517231 podStartE2EDuration="2.51316597s" totalImagesPullingTime="601.648739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:15:57.51316346 +0000 UTC m=+1639.363923291" watchObservedRunningTime="2026-07-21 22:15:57.51316597 +0000 UTC m=+1639.363925802" Jul 21 22:16:03.501622 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:03.501578 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:16:21.192325 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.192296 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248"] Jul 21 22:16:21.195517 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.195501 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:21.198525 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.198508 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:16:21.207014 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.206994 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248"] Jul 21 22:16:21.294533 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.294504 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-678339c2-sc248\" (UID: \"35610632-2da2-49ac-9f19-7141a80323ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:21.395443 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.395407 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-678339c2-sc248\" (UID: \"35610632-2da2-49ac-9f19-7141a80323ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:21.396101 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.396080 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-678339c2-sc248\" (UID: \"35610632-2da2-49ac-9f19-7141a80323ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:21.506718 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.506661 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:21.635831 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:21.635801 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248"] Jul 21 22:16:21.639015 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:16:21.638984 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35610632_2da2_49ac_9f19_7141a80323ec.slice/crio-83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff WatchSource:0}: Error finding container 83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff: Status 404 returned error can't find the container with id 83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff Jul 21 22:16:22.604198 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:22.604165 2555 generic.go:356] "Generic (PLEG): container finished" podID="35610632-2da2-49ac-9f19-7141a80323ec" containerID="aa1e20609dab0d4d1f1b6653675852cd5e194684f657db22fa1a209d5c0e633f" exitCode=0 Jul 21 22:16:22.604595 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:22.604255 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" event={"ID":"35610632-2da2-49ac-9f19-7141a80323ec","Type":"ContainerDied","Data":"aa1e20609dab0d4d1f1b6653675852cd5e194684f657db22fa1a209d5c0e633f"} Jul 21 22:16:22.604595 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:22.604313 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" event={"ID":"35610632-2da2-49ac-9f19-7141a80323ec","Type":"ContainerStarted","Data":"83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff"} Jul 21 22:16:23.739145 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:23.739124 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:23.912850 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:23.912788 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca\") pod \"35610632-2da2-49ac-9f19-7141a80323ec\" (UID: \"35610632-2da2-49ac-9f19-7141a80323ec\") " Jul 21 22:16:23.913156 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:23.913131 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35610632-2da2-49ac-9f19-7141a80323ec" (UID: "35610632-2da2-49ac-9f19-7141a80323ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:16:24.014172 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:24.014139 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35610632-2da2-49ac-9f19-7141a80323ec-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:16:24.615910 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:24.615880 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" Jul 21 22:16:24.616077 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:24.615888 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248" event={"ID":"35610632-2da2-49ac-9f19-7141a80323ec","Type":"ContainerDied","Data":"83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff"} Jul 21 22:16:24.616077 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:24.616032 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83b22c4c919f59b781fb1d36f8f1f15c5afcdce107203149cdfa8f0ecc305bff" Jul 21 22:16:31.899914 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:31.899878 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:16:31.900455 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:31.900092 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" podUID="a04a085f-7038-4b59-b05b-289220dec8d7" containerName="maas-api" containerID="cri-o://4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0" gracePeriod=30 Jul 21 22:16:31.931520 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:31.931496 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:32.146812 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.146783 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:16:32.274395 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.274323 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca\") pod \"a04a085f-7038-4b59-b05b-289220dec8d7\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " Jul 21 22:16:32.274537 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.274408 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls\") pod \"a04a085f-7038-4b59-b05b-289220dec8d7\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " Jul 21 22:16:32.274537 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.274435 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjw7l\" (UniqueName: \"kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l\") pod \"a04a085f-7038-4b59-b05b-289220dec8d7\" (UID: \"a04a085f-7038-4b59-b05b-289220dec8d7\") " Jul 21 22:16:32.274673 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.274631 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a04a085f-7038-4b59-b05b-289220dec8d7" (UID: "a04a085f-7038-4b59-b05b-289220dec8d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:16:32.276491 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.276464 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l" (OuterVolumeSpecName: "kube-api-access-bjw7l") pod "a04a085f-7038-4b59-b05b-289220dec8d7" (UID: "a04a085f-7038-4b59-b05b-289220dec8d7"). InnerVolumeSpecName "kube-api-access-bjw7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:16:32.276491 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.276480 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a04a085f-7038-4b59-b05b-289220dec8d7" (UID: "a04a085f-7038-4b59-b05b-289220dec8d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:16:32.376025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.375998 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a04a085f-7038-4b59-b05b-289220dec8d7-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:16:32.376025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.376020 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a04a085f-7038-4b59-b05b-289220dec8d7-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:16:32.376025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.376030 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjw7l\" (UniqueName: \"kubernetes.io/projected/a04a085f-7038-4b59-b05b-289220dec8d7-kube-api-access-bjw7l\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:16:32.650742 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.650712 2555 generic.go:356] "Generic (PLEG): container finished" podID="a04a085f-7038-4b59-b05b-289220dec8d7" containerID="4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0" exitCode=0 Jul 21 22:16:32.650872 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.650750 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" event={"ID":"a04a085f-7038-4b59-b05b-289220dec8d7","Type":"ContainerDied","Data":"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0"} Jul 21 22:16:32.650872 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.650756 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" Jul 21 22:16:32.650872 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.650786 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg" event={"ID":"a04a085f-7038-4b59-b05b-289220dec8d7","Type":"ContainerDied","Data":"7b5a76571028cd69d8556886db9567b262c5502417c533883190a55b0d37f2d7"} Jul 21 22:16:32.650872 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.650807 2555 scope.go:117] "RemoveContainer" containerID="4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0" Jul 21 22:16:32.659942 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.659925 2555 scope.go:117] "RemoveContainer" containerID="4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0" Jul 21 22:16:32.660202 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:16:32.660183 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0\": container with ID starting with 4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0 not found: ID does not exist" containerID="4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0" Jul 21 22:16:32.660259 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.660210 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0"} err="failed to get container status \"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0\": rpc error: code = NotFound desc = could not find container \"4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0\": container with ID starting with 4a570346bc90ea05aa086654cd48ff7a2199c3ca39c7778109cf7dad9a7a3ce0 not found: ID does not exist" Jul 21 22:16:32.675686 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.675661 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:16:32.679953 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:32.679932 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-678339c2-5c9959cfd8-wr4pg"] Jul 21 22:16:34.664251 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:34.664208 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04a085f-7038-4b59-b05b-289220dec8d7" path="/var/lib/kubelet/pods/a04a085f-7038-4b59-b05b-289220dec8d7/volumes" Jul 21 22:16:36.277431 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.277398 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248"] Jul 21 22:16:36.278837 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.278807 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:36.280502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.280482 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-678339c2-sc248"] Jul 21 22:16:36.435471 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.435441 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:36.664455 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.664425 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35610632-2da2-49ac-9f19-7141a80323ec" path="/var/lib/kubelet/pods/35610632-2da2-49ac-9f19-7141a80323ec/volumes" Jul 21 22:16:36.682424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.682396 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:36.939346 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:36.939269 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:37.029707 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:37.029674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:37.153016 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:37.152988 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:38.123630 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:38.123576 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:38.186759 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:38.186733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:39.982048 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:39.982006 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:40.081102 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081071 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:16:40.081440 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081427 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04a085f-7038-4b59-b05b-289220dec8d7" containerName="maas-api" Jul 21 22:16:40.081487 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081443 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04a085f-7038-4b59-b05b-289220dec8d7" containerName="maas-api" Jul 21 22:16:40.081487 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081454 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35610632-2da2-49ac-9f19-7141a80323ec" containerName="revoke-keys" Jul 21 22:16:40.081487 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081461 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="35610632-2da2-49ac-9f19-7141a80323ec" containerName="revoke-keys" Jul 21 22:16:40.081581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081523 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="35610632-2da2-49ac-9f19-7141a80323ec" containerName="revoke-keys" Jul 21 22:16:40.081581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.081533 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04a085f-7038-4b59-b05b-289220dec8d7" containerName="maas-api" Jul 21 22:16:40.084183 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.084163 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.087179 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.087157 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-dd422c5b-serving-cert\"" Jul 21 22:16:40.088414 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.088398 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:16:40.095976 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.095949 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:16:40.233896 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.233822 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnmq\" (UniqueName: \"kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.233896 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.233862 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.233896 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.233886 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.334783 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.334752 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnmq\" (UniqueName: \"kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.334897 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.334788 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.334897 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.334814 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.335454 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.335430 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.337022 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.336996 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.345533 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.345509 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnmq\" (UniqueName: \"kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq\") pod \"maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.395331 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.395309 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:40.733144 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:40.733123 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:16:40.734793 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:16:40.734769 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b14c17f_2d7b_49b2_a524_ca00fddb6cee.slice/crio-503f0d7b1753e84ffcd10a51255153c7fd22cfbfac17adc6b250b1b119ae1942 WatchSource:0}: Error finding container 503f0d7b1753e84ffcd10a51255153c7fd22cfbfac17adc6b250b1b119ae1942: Status 404 returned error can't find the container with id 503f0d7b1753e84ffcd10a51255153c7fd22cfbfac17adc6b250b1b119ae1942 Jul 21 22:16:41.690646 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:41.690596 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" event={"ID":"2b14c17f-2d7b-49b2-a524-ca00fddb6cee","Type":"ContainerStarted","Data":"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff"} Jul 21 22:16:41.690983 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:41.690659 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" event={"ID":"2b14c17f-2d7b-49b2-a524-ca00fddb6cee","Type":"ContainerStarted","Data":"503f0d7b1753e84ffcd10a51255153c7fd22cfbfac17adc6b250b1b119ae1942"} Jul 21 22:16:41.690983 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:41.690715 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:16:41.709843 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:41.709801 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" podStartSLOduration=0.944423872 podStartE2EDuration="1.709787158s" totalImagesPullingTime="765.363286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:16:41.707993809 +0000 UTC m=+1683.558753641" watchObservedRunningTime="2026-07-21 22:16:41.709787158 +0000 UTC m=+1683.560546989" Jul 21 22:16:42.091523 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:42.091493 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:16:47.700372 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:16:47.700342 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:17:04.952905 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:04.952804 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84"] Jul 21 22:17:04.956697 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:04.956674 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:04.959495 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:04.959478 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:17:04.964801 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:04.964778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84"] Jul 21 22:17:05.012656 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.012631 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84\" (UID: \"c4ad9be0-af48-448d-8540-ab8025e4dbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:05.113529 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.113504 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84\" (UID: \"c4ad9be0-af48-448d-8540-ab8025e4dbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:05.114150 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.114128 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84\" (UID: \"c4ad9be0-af48-448d-8540-ab8025e4dbb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:05.268233 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.268155 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:05.396319 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.396294 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84"] Jul 21 22:17:05.398503 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:17:05.398478 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ad9be0_af48_448d_8540_ab8025e4dbb4.slice/crio-f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52 WatchSource:0}: Error finding container f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52: Status 404 returned error can't find the container with id f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52 Jul 21 22:17:05.797059 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.797027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" event={"ID":"c4ad9be0-af48-448d-8540-ab8025e4dbb4","Type":"ContainerStarted","Data":"3641f5ad9747e24d2704f61e19b0b0b0f2774501ee23843dd41cab83457387f2"} Jul 21 22:17:05.797187 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.797082 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" event={"ID":"c4ad9be0-af48-448d-8540-ab8025e4dbb4","Type":"ContainerStarted","Data":"f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52"} Jul 21 22:17:05.813803 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:05.813760 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" podStartSLOduration=1.813746131 podStartE2EDuration="1.813746131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:17:05.812557177 +0000 UTC m=+1707.663317007" watchObservedRunningTime="2026-07-21 22:17:05.813746131 +0000 UTC m=+1707.664505961" Jul 21 22:17:06.802210 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:06.802175 2555 generic.go:356] "Generic (PLEG): container finished" podID="c4ad9be0-af48-448d-8540-ab8025e4dbb4" containerID="3641f5ad9747e24d2704f61e19b0b0b0f2774501ee23843dd41cab83457387f2" exitCode=0 Jul 21 22:17:06.802621 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:06.802233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" event={"ID":"c4ad9be0-af48-448d-8540-ab8025e4dbb4","Type":"ContainerDied","Data":"3641f5ad9747e24d2704f61e19b0b0b0f2774501ee23843dd41cab83457387f2"} Jul 21 22:17:07.944472 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:07.944445 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:08.035992 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.035967 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca\") pod \"c4ad9be0-af48-448d-8540-ab8025e4dbb4\" (UID: \"c4ad9be0-af48-448d-8540-ab8025e4dbb4\") " Jul 21 22:17:08.036274 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.036253 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4ad9be0-af48-448d-8540-ab8025e4dbb4" (UID: "c4ad9be0-af48-448d-8540-ab8025e4dbb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:17:08.136651 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.136629 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ad9be0-af48-448d-8540-ab8025e4dbb4-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:08.814202 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.814175 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" Jul 21 22:17:08.814202 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.814171 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84" event={"ID":"c4ad9be0-af48-448d-8540-ab8025e4dbb4","Type":"ContainerDied","Data":"f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52"} Jul 21 22:17:08.814436 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:08.814219 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f951b2bf576b5a0f6e466f16d6bf4bab0903c6f17731ceda74d0e56bea3bdc52" Jul 21 22:17:15.417210 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.417174 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:17:15.417736 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.417440 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" podUID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" containerName="maas-api" containerID="cri-o://cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff" gracePeriod=30 Jul 21 22:17:15.433819 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.433792 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:15.680743 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.680687 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:17:15.805025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.804999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca\") pod \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " Jul 21 22:17:15.805160 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.805033 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls\") pod \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " Jul 21 22:17:15.805160 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.805072 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfnmq\" (UniqueName: \"kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq\") pod \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\" (UID: \"2b14c17f-2d7b-49b2-a524-ca00fddb6cee\") " Jul 21 22:17:15.805450 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.805419 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b14c17f-2d7b-49b2-a524-ca00fddb6cee" (UID: "2b14c17f-2d7b-49b2-a524-ca00fddb6cee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:17:15.807130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.807098 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq" (OuterVolumeSpecName: "kube-api-access-kfnmq") pod "2b14c17f-2d7b-49b2-a524-ca00fddb6cee" (UID: "2b14c17f-2d7b-49b2-a524-ca00fddb6cee"). InnerVolumeSpecName "kube-api-access-kfnmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:17:15.807209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.807096 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b14c17f-2d7b-49b2-a524-ca00fddb6cee" (UID: "2b14c17f-2d7b-49b2-a524-ca00fddb6cee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:17:15.846914 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.846887 2555 generic.go:356] "Generic (PLEG): container finished" podID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" containerID="cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff" exitCode=0 Jul 21 22:17:15.847007 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.846929 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" Jul 21 22:17:15.847007 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.846971 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" event={"ID":"2b14c17f-2d7b-49b2-a524-ca00fddb6cee","Type":"ContainerDied","Data":"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff"} Jul 21 22:17:15.847091 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.847043 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27" event={"ID":"2b14c17f-2d7b-49b2-a524-ca00fddb6cee","Type":"ContainerDied","Data":"503f0d7b1753e84ffcd10a51255153c7fd22cfbfac17adc6b250b1b119ae1942"} Jul 21 22:17:15.847091 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.847068 2555 scope.go:117] "RemoveContainer" containerID="cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff" Jul 21 22:17:15.855472 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.855458 2555 scope.go:117] "RemoveContainer" containerID="cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff" Jul 21 22:17:15.855739 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:17:15.855722 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff\": container with ID starting with cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff not found: ID does not exist" containerID="cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff" Jul 21 22:17:15.855792 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.855745 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff"} err="failed to get container status \"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff\": rpc error: code = NotFound desc = could not find container \"cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff\": container with ID starting with cf582f133960de471f252507246c1e65562edfca5f90c96440a54fc80f1703ff not found: ID does not exist" Jul 21 22:17:15.867987 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.867962 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:17:15.874245 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.874220 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-dd422c5b-794859f8c6-2gp27"] Jul 21 22:17:15.906263 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.906232 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnmq\" (UniqueName: \"kubernetes.io/projected/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-kube-api-access-kfnmq\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:15.906263 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.906254 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:15.906380 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:15.906269 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b14c17f-2d7b-49b2-a524-ca00fddb6cee-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:16.667581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:16.667545 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" path="/var/lib/kubelet/pods/2b14c17f-2d7b-49b2-a524-ca00fddb6cee/volumes" Jul 21 22:17:20.027268 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.027239 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:20.031283 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.031258 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84"] Jul 21 22:17:20.033631 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.033592 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd422c5b-hjb84"] Jul 21 22:17:20.229233 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.229197 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:20.431030 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.430999 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:20.664157 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.664129 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ad9be0-af48-448d-8540-ab8025e4dbb4" path="/var/lib/kubelet/pods/c4ad9be0-af48-448d-8540-ab8025e4dbb4/volumes" Jul 21 22:17:20.675879 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:20.675853 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:21.004527 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:21.004497 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:21.689873 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:21.689846 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.117064 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.117015 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.222110 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.222058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.276502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276456 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:17:23.276862 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276845 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ad9be0-af48-448d-8540-ab8025e4dbb4" containerName="revoke-keys" Jul 21 22:17:23.276926 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276863 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ad9be0-af48-448d-8540-ab8025e4dbb4" containerName="revoke-keys" Jul 21 22:17:23.276926 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276875 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" containerName="maas-api" Jul 21 22:17:23.276926 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276881 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" containerName="maas-api" Jul 21 22:17:23.277017 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276945 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ad9be0-af48-448d-8540-ab8025e4dbb4" containerName="revoke-keys" Jul 21 22:17:23.277017 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.276954 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b14c17f-2d7b-49b2-a524-ca00fddb6cee" containerName="maas-api" Jul 21 22:17:23.281162 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.281142 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.288281 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.288264 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:17:23.288367 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.288340 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bc2a2d3f-serving-cert\"" Jul 21 22:17:23.296598 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.296575 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:17:23.375068 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.375012 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.375068 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.375042 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fc24\" (UniqueName: \"kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.375206 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.375074 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.475678 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.475643 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.475794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.475683 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fc24\" (UniqueName: \"kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.475794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.475715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.475904 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:17:23.475829 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-serving-cert: secret "maas-api-e2e-ait-bc2a2d3f-serving-cert" not found Jul 21 22:17:23.475904 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:17:23.475889 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls podName:f9321e93-a535-490c-83d6-ff461c5dea56 nodeName:}" failed. No retries permitted until 2026-07-21 22:17:23.975874411 +0000 UTC m=+1725.826634220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls") pod "maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" (UID: "f9321e93-a535-490c-83d6-ff461c5dea56") : secret "maas-api-e2e-ait-bc2a2d3f-serving-cert" not found Jul 21 22:17:23.476255 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.476235 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.507993 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.507971 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fc24\" (UniqueName: \"kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.573347 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.573328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.672601 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.672530 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.945297 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.945223 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:23.980093 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.980063 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:23.982399 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:23.982366 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") pod \"maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:24.047376 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:24.047350 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:24.191112 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:24.191085 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:24.318864 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:24.318828 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:17:24.321645 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:17:24.321591 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9321e93_a535_490c_83d6_ff461c5dea56.slice/crio-4f2312f4d887189835dbb37d085c7422c061af902adb1c04b01cd0799934634a WatchSource:0}: Error finding container 4f2312f4d887189835dbb37d085c7422c061af902adb1c04b01cd0799934634a: Status 404 returned error can't find the container with id 4f2312f4d887189835dbb37d085c7422c061af902adb1c04b01cd0799934634a Jul 21 22:17:24.887529 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:24.887495 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" event={"ID":"f9321e93-a535-490c-83d6-ff461c5dea56","Type":"ContainerStarted","Data":"4f2312f4d887189835dbb37d085c7422c061af902adb1c04b01cd0799934634a"} Jul 21 22:17:25.893271 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:25.893233 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" event={"ID":"f9321e93-a535-490c-83d6-ff461c5dea56","Type":"ContainerStarted","Data":"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35"} Jul 21 22:17:25.893682 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:25.893387 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:25.911305 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:25.911253 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" podStartSLOduration=2.258109995 podStartE2EDuration="2.911237703s" totalImagesPullingTime="653.127708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:17:25.9097666 +0000 UTC m=+1727.760526433" watchObservedRunningTime="2026-07-21 22:17:25.911237703 +0000 UTC m=+1727.761997533" Jul 21 22:17:31.903375 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:31.903344 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:48.921724 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:48.921686 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd"] Jul 21 22:17:48.925061 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:48.925042 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:48.927597 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:48.927575 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:17:48.933774 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:48.933751 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd"] Jul 21 22:17:48.956634 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:48.956594 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd\" (UID: \"24ccb1de-6b64-4ef5-af0c-f25c928151a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:49.057190 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.057166 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd\" (UID: \"24ccb1de-6b64-4ef5-af0c-f25c928151a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:49.058054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.058028 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd\" (UID: \"24ccb1de-6b64-4ef5-af0c-f25c928151a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:49.236545 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.236481 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:49.364737 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.364711 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd"] Jul 21 22:17:49.366555 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:17:49.366531 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ccb1de_6b64_4ef5_af0c_f25c928151a3.slice/crio-4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587 WatchSource:0}: Error finding container 4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587: Status 404 returned error can't find the container with id 4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587 Jul 21 22:17:49.997876 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.997841 2555 generic.go:356] "Generic (PLEG): container finished" podID="24ccb1de-6b64-4ef5-af0c-f25c928151a3" containerID="b223fa255c2f44a381c78075020c4805c901dd15eaf5101400b0ae24d1952aa4" exitCode=0 Jul 21 22:17:49.998179 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.997914 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" event={"ID":"24ccb1de-6b64-4ef5-af0c-f25c928151a3","Type":"ContainerDied","Data":"b223fa255c2f44a381c78075020c4805c901dd15eaf5101400b0ae24d1952aa4"} Jul 21 22:17:49.998179 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:49.997969 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" event={"ID":"24ccb1de-6b64-4ef5-af0c-f25c928151a3","Type":"ContainerStarted","Data":"4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587"} Jul 21 22:17:51.135400 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:51.135376 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:51.173208 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:51.173182 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca\") pod \"24ccb1de-6b64-4ef5-af0c-f25c928151a3\" (UID: \"24ccb1de-6b64-4ef5-af0c-f25c928151a3\") " Jul 21 22:17:51.173523 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:51.173501 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24ccb1de-6b64-4ef5-af0c-f25c928151a3" (UID: "24ccb1de-6b64-4ef5-af0c-f25c928151a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:17:51.273783 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:51.273727 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ccb1de-6b64-4ef5-af0c-f25c928151a3-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:52.013161 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:52.013129 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" Jul 21 22:17:52.013337 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:52.013128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd" event={"ID":"24ccb1de-6b64-4ef5-af0c-f25c928151a3","Type":"ContainerDied","Data":"4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587"} Jul 21 22:17:52.013337 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:52.013243 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba75597f0612dbb3e79afc79e57439c792746b2325f69f22f754d708d2c9587" Jul 21 22:17:59.414901 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.414868 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:17:59.415363 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.415080 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" podUID="f9321e93-a535-490c-83d6-ff461c5dea56" containerName="maas-api" containerID="cri-o://c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35" gracePeriod=30 Jul 21 22:17:59.437797 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.437771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:17:59.661209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.661189 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:17:59.732922 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.732846 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca\") pod \"f9321e93-a535-490c-83d6-ff461c5dea56\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " Jul 21 22:17:59.732922 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.732875 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fc24\" (UniqueName: \"kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24\") pod \"f9321e93-a535-490c-83d6-ff461c5dea56\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " Jul 21 22:17:59.732922 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.732910 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") pod \"f9321e93-a535-490c-83d6-ff461c5dea56\" (UID: \"f9321e93-a535-490c-83d6-ff461c5dea56\") " Jul 21 22:17:59.733253 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.733227 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9321e93-a535-490c-83d6-ff461c5dea56" (UID: "f9321e93-a535-490c-83d6-ff461c5dea56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:17:59.734957 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.734935 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9321e93-a535-490c-83d6-ff461c5dea56" (UID: "f9321e93-a535-490c-83d6-ff461c5dea56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:17:59.735036 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.735018 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24" (OuterVolumeSpecName: "kube-api-access-9fc24") pod "f9321e93-a535-490c-83d6-ff461c5dea56" (UID: "f9321e93-a535-490c-83d6-ff461c5dea56"). InnerVolumeSpecName "kube-api-access-9fc24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:17:59.833428 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.833405 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9321e93-a535-490c-83d6-ff461c5dea56-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:59.833428 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.833428 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fc24\" (UniqueName: \"kubernetes.io/projected/f9321e93-a535-490c-83d6-ff461c5dea56-kube-api-access-9fc24\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:17:59.833556 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:17:59.833437 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9321e93-a535-490c-83d6-ff461c5dea56-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:18:00.050884 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.050813 2555 generic.go:356] "Generic (PLEG): container finished" podID="f9321e93-a535-490c-83d6-ff461c5dea56" containerID="c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35" exitCode=0 Jul 21 22:18:00.050884 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.050864 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" Jul 21 22:18:00.051019 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.050896 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" event={"ID":"f9321e93-a535-490c-83d6-ff461c5dea56","Type":"ContainerDied","Data":"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35"} Jul 21 22:18:00.051019 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.050949 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn" event={"ID":"f9321e93-a535-490c-83d6-ff461c5dea56","Type":"ContainerDied","Data":"4f2312f4d887189835dbb37d085c7422c061af902adb1c04b01cd0799934634a"} Jul 21 22:18:00.051019 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.050972 2555 scope.go:117] "RemoveContainer" containerID="c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35" Jul 21 22:18:00.059215 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.059198 2555 scope.go:117] "RemoveContainer" containerID="c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35" Jul 21 22:18:00.059459 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:18:00.059439 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35\": container with ID starting with c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35 not found: ID does not exist" containerID="c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35" Jul 21 22:18:00.059502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.059468 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35"} err="failed to get container status \"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35\": rpc error: code = NotFound desc = could not find container \"c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35\": container with ID starting with c0dd1c9dcff03f687c70d5c306f1d1e68859295bfb374cc9126ba07393bddb35 not found: ID does not exist" Jul 21 22:18:00.072169 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.072150 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:18:00.076114 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.076096 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bc2a2d3f-c44f7c7cd-4vxpn"] Jul 21 22:18:00.664474 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:00.664444 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9321e93-a535-490c-83d6-ff461c5dea56" path="/var/lib/kubelet/pods/f9321e93-a535-490c-83d6-ff461c5dea56/volumes" Jul 21 22:18:04.017090 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:04.017047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:04.022632 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:04.022597 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd"] Jul 21 22:18:04.024701 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:04.024679 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc2a2d3f-q9twd"] Jul 21 22:18:04.664546 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:04.664513 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ccb1de-6b64-4ef5-af0c-f25c928151a3" path="/var/lib/kubelet/pods/24ccb1de-6b64-4ef5-af0c-f25c928151a3/volumes" Jul 21 22:18:05.225699 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:05.225663 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:05.697560 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:05.697526 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:05.798627 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:05.798576 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:05.978539 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:05.978449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:06.936548 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:06.936512 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:08.397502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.397460 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:08.494825 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.494798 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:08.649520 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.649448 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:08.649953 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.649937 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9321e93-a535-490c-83d6-ff461c5dea56" containerName="maas-api" Jul 21 22:18:08.649998 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.649957 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9321e93-a535-490c-83d6-ff461c5dea56" containerName="maas-api" Jul 21 22:18:08.649998 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.649978 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ccb1de-6b64-4ef5-af0c-f25c928151a3" containerName="revoke-keys" Jul 21 22:18:08.649998 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.649987 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ccb1de-6b64-4ef5-af0c-f25c928151a3" containerName="revoke-keys" Jul 21 22:18:08.650094 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.650063 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ccb1de-6b64-4ef5-af0c-f25c928151a3" containerName="revoke-keys" Jul 21 22:18:08.650094 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.650078 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9321e93-a535-490c-83d6-ff461c5dea56" containerName="maas-api" Jul 21 22:18:08.654642 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.654622 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.657159 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.657139 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:18:08.657377 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.657360 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-85902df0-serving-cert\"" Jul 21 22:18:08.666831 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.666805 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:08.685303 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.685273 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:08.700735 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.700711 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.700826 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.700746 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbhh\" (UniqueName: \"kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.700826 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.700770 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.801168 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.801143 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbhh\" (UniqueName: \"kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.801278 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.801188 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.801339 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.801277 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.801394 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:18:08.801376 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-serving-cert: secret "maas-api-e2e-derive-85902df0-serving-cert" not found Jul 21 22:18:08.801457 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:18:08.801446 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls podName:454f2dc0-1ef9-46aa-b8a5-c475ac9743c2 nodeName:}" failed. No retries permitted until 2026-07-21 22:18:09.301427007 +0000 UTC m=+1771.152186820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls") pod "maas-api-e2e-derive-85902df0-cbf67546-2mndk" (UID: "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2") : secret "maas-api-e2e-derive-85902df0-serving-cert" not found Jul 21 22:18:08.801871 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.801851 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:08.809960 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:08.809939 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbhh\" (UniqueName: \"kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:09.141810 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.141761 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:09.245774 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.245730 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:09.305499 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.305470 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:09.308042 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.308012 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") pod \"maas-api-e2e-derive-85902df0-cbf67546-2mndk\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:09.566026 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.565955 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:09.696840 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:09.696817 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:09.699294 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:18:09.699264 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454f2dc0_1ef9_46aa_b8a5_c475ac9743c2.slice/crio-2fb60badd5f43e361b1b08358d4cf872af77b1456d9d1893321b9f1c61fe4441 WatchSource:0}: Error finding container 2fb60badd5f43e361b1b08358d4cf872af77b1456d9d1893321b9f1c61fe4441: Status 404 returned error can't find the container with id 2fb60badd5f43e361b1b08358d4cf872af77b1456d9d1893321b9f1c61fe4441 Jul 21 22:18:10.099265 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:10.099236 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" event={"ID":"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2","Type":"ContainerStarted","Data":"2fb60badd5f43e361b1b08358d4cf872af77b1456d9d1893321b9f1c61fe4441"} Jul 21 22:18:11.105539 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:11.105500 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" event={"ID":"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2","Type":"ContainerStarted","Data":"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92"} Jul 21 22:18:11.106004 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:11.105662 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:11.125179 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:11.125137 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" podStartSLOduration=2.761138937 podStartE2EDuration="3.125124137s" totalImagesPullingTime="363.9852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:18:11.12276948 +0000 UTC m=+1772.973529312" watchObservedRunningTime="2026-07-21 22:18:11.125124137 +0000 UTC m=+1772.975883968" Jul 21 22:18:17.114964 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:17.114939 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:20.575271 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:20.575221 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:20.675887 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:20.675843 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:29.911359 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:29.911277 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:30.014159 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:30.014121 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:33.223853 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.223819 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q"] Jul 21 22:18:33.227387 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.227372 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:33.229975 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.229956 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:18:33.234600 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.234576 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q"] Jul 21 22:18:33.287015 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.286981 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85902df0-tl82q\" (UID: \"70144c12-6597-4b2e-91ee-91fc772340a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:33.387359 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.387325 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85902df0-tl82q\" (UID: \"70144c12-6597-4b2e-91ee-91fc772340a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:33.388020 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.387994 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85902df0-tl82q\" (UID: \"70144c12-6597-4b2e-91ee-91fc772340a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:33.538141 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.538063 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:33.664937 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:33.664872 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q"] Jul 21 22:18:33.667089 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:18:33.667054 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70144c12_6597_4b2e_91ee_91fc772340a1.slice/crio-a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45 WatchSource:0}: Error finding container a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45: Status 404 returned error can't find the container with id a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45 Jul 21 22:18:34.217727 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:34.217635 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" event={"ID":"70144c12-6597-4b2e-91ee-91fc772340a1","Type":"ContainerStarted","Data":"fb5800511735b660625bd77a2ba45a85119fcf618d7c25988ac1eeaac83cad65"} Jul 21 22:18:34.217727 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:34.217697 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" event={"ID":"70144c12-6597-4b2e-91ee-91fc772340a1","Type":"ContainerStarted","Data":"a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45"} Jul 21 22:18:34.233025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:34.232976 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" podStartSLOduration=1.232960051 podStartE2EDuration="1.232960051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:18:34.232662467 +0000 UTC m=+1796.083422298" watchObservedRunningTime="2026-07-21 22:18:34.232960051 +0000 UTC m=+1796.083719926" Jul 21 22:18:35.223763 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:35.223727 2555 generic.go:356] "Generic (PLEG): container finished" podID="70144c12-6597-4b2e-91ee-91fc772340a1" containerID="fb5800511735b660625bd77a2ba45a85119fcf618d7c25988ac1eeaac83cad65" exitCode=0 Jul 21 22:18:35.223955 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:35.223792 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" event={"ID":"70144c12-6597-4b2e-91ee-91fc772340a1","Type":"ContainerDied","Data":"fb5800511735b660625bd77a2ba45a85119fcf618d7c25988ac1eeaac83cad65"} Jul 21 22:18:36.355002 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:36.354978 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:36.411597 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:36.411570 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca\") pod \"70144c12-6597-4b2e-91ee-91fc772340a1\" (UID: \"70144c12-6597-4b2e-91ee-91fc772340a1\") " Jul 21 22:18:36.411939 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:36.411917 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70144c12-6597-4b2e-91ee-91fc772340a1" (UID: "70144c12-6597-4b2e-91ee-91fc772340a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:18:36.512353 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:36.512273 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70144c12-6597-4b2e-91ee-91fc772340a1-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:18:37.234729 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:37.234693 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" event={"ID":"70144c12-6597-4b2e-91ee-91fc772340a1","Type":"ContainerDied","Data":"a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45"} Jul 21 22:18:37.234729 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:37.234709 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q" Jul 21 22:18:37.234952 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:37.234742 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19afb22a59f02131dd9dfd4f81199dd0deba4e5651d440a92f0e21e1b499f45" Jul 21 22:18:43.724831 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:43.724799 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:43.725238 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:43.725038 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" podUID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" containerName="maas-api" containerID="cri-o://c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92" gracePeriod=30 Jul 21 22:18:43.748796 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:43.748763 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:43.973387 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:43.973364 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:44.071050 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.071028 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca\") pod \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " Jul 21 22:18:44.071205 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.071058 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jbhh\" (UniqueName: \"kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh\") pod \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " Jul 21 22:18:44.071205 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.071118 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") pod \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\" (UID: \"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2\") " Jul 21 22:18:44.071384 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.071359 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" (UID: "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:18:44.073187 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.073152 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" (UID: "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:18:44.073296 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.073202 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh" (OuterVolumeSpecName: "kube-api-access-6jbhh") pod "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" (UID: "454f2dc0-1ef9-46aa-b8a5-c475ac9743c2"). InnerVolumeSpecName "kube-api-access-6jbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:18:44.172424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.172393 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:18:44.172424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.172424 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:18:44.172650 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.172436 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbhh\" (UniqueName: \"kubernetes.io/projected/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2-kube-api-access-6jbhh\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:18:44.268470 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.268433 2555 generic.go:356] "Generic (PLEG): container finished" podID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" containerID="c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92" exitCode=0 Jul 21 22:18:44.268662 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.268480 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" Jul 21 22:18:44.268662 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.268492 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" event={"ID":"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2","Type":"ContainerDied","Data":"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92"} Jul 21 22:18:44.268662 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.268547 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk" event={"ID":"454f2dc0-1ef9-46aa-b8a5-c475ac9743c2","Type":"ContainerDied","Data":"2fb60badd5f43e361b1b08358d4cf872af77b1456d9d1893321b9f1c61fe4441"} Jul 21 22:18:44.268662 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.268577 2555 scope.go:117] "RemoveContainer" containerID="c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92" Jul 21 22:18:44.277579 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.277560 2555 scope.go:117] "RemoveContainer" containerID="c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92" Jul 21 22:18:44.277898 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:18:44.277879 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92\": container with ID starting with c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92 not found: ID does not exist" containerID="c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92" Jul 21 22:18:44.277952 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.277908 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92"} err="failed to get container status \"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92\": rpc error: code = NotFound desc = could not find container \"c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92\": container with ID starting with c6a53a50eeef78c20e17d48776562797827479c74277dc2ede50d236ad6eda92 not found: ID does not exist" Jul 21 22:18:44.290575 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.290550 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:44.295345 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.295323 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85902df0-cbf67546-2mndk"] Jul 21 22:18:44.664482 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:44.664451 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" path="/var/lib/kubelet/pods/454f2dc0-1ef9-46aa-b8a5-c475ac9743c2/volumes" Jul 21 22:18:48.305487 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.305452 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:48.312589 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.312565 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q"] Jul 21 22:18:48.315087 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.315066 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85902df0-tl82q"] Jul 21 22:18:48.467298 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.467263 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:48.664729 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.664699 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70144c12-6597-4b2e-91ee-91fc772340a1" path="/var/lib/kubelet/pods/70144c12-6597-4b2e-91ee-91fc772340a1/volumes" Jul 21 22:18:48.704849 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:48.704821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:49.196016 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:49.195982 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:49.291888 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:49.291859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:49.394241 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:49.394210 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:50.836813 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:50.836773 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:57.694882 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:57.694849 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:57.819754 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:57.819723 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:57.916892 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:57.916859 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:58.003118 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:58.003037 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:18:58.484189 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:18:58.484151 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:24.445275 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:24.445242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:30.832402 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:30.832373 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:30.934498 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:30.934465 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:31.032528 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:31.032485 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:46.967568 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:46.967531 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:52.901698 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:52.901653 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:52.959721 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:52.959686 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:53.038988 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:53.038952 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:19:53.137686 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:19:53.137658 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:03.637817 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:03.637744 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:04.479267 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:04.479233 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:11.418273 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:11.418242 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:11.531521 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:11.531488 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:11.618029 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:11.617995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:11.786893 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:11.786788 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:12.542384 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:12.542338 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:12.636221 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:12.636181 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:12.737595 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:12.737561 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:14.367161 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:14.367123 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:20.329022 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:20.328991 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:20.430794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:20.430765 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:21.111027 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:21.110995 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:27.407983 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:27.407941 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:27.470369 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:27.470342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:27.662479 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:27.662405 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:28.022130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:28.022047 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:28.319624 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:28.319576 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:30.914870 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:30.914832 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:30.997331 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:30.997300 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:31.134984 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.134950 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:31.135349 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135331 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70144c12-6597-4b2e-91ee-91fc772340a1" containerName="revoke-keys" Jul 21 22:20:31.135439 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135352 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70144c12-6597-4b2e-91ee-91fc772340a1" containerName="revoke-keys" Jul 21 22:20:31.135439 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135375 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" containerName="maas-api" Jul 21 22:20:31.135439 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135384 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" containerName="maas-api" Jul 21 22:20:31.135591 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135485 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="454f2dc0-1ef9-46aa-b8a5-c475ac9743c2" containerName="maas-api" Jul 21 22:20:31.135591 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.135499 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="70144c12-6597-4b2e-91ee-91fc772340a1" containerName="revoke-keys" Jul 21 22:20:31.138364 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.138342 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.141481 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.141436 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:20:31.142560 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.142544 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e0aec917-serving-cert\"" Jul 21 22:20:31.143104 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.143082 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:31.165514 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.165449 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:31.187902 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.187876 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.187991 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.187910 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmqv\" (UniqueName: \"kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.187991 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.187948 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.268805 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.268764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:31.288288 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.288267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmqv\" (UniqueName: \"kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.288373 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.288314 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.288373 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.288366 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.288986 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.288963 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.291014 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.290996 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.296913 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.296883 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmqv\" (UniqueName: \"kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv\") pod \"maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.451433 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.451362 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:31.687363 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.687331 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:31.787710 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.787665 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:31.789085 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:31.789058 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:31.789912 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:20:31.789891 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9c2dbf_5882_4bd7_a3fc_06fc9e006e89.slice/crio-c4cf3c62bbdb0cfadccd67ce51dcc37ad4443c9838f3dc9675f24cc0c09f462d WatchSource:0}: Error finding container c4cf3c62bbdb0cfadccd67ce51dcc37ad4443c9838f3dc9675f24cc0c09f462d: Status 404 returned error can't find the container with id c4cf3c62bbdb0cfadccd67ce51dcc37ad4443c9838f3dc9675f24cc0c09f462d Jul 21 22:20:32.757727 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:32.757696 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" event={"ID":"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89","Type":"ContainerStarted","Data":"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff"} Jul 21 22:20:32.758126 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:32.757739 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" event={"ID":"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89","Type":"ContainerStarted","Data":"c4cf3c62bbdb0cfadccd67ce51dcc37ad4443c9838f3dc9675f24cc0c09f462d"} Jul 21 22:20:32.758126 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:32.757816 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:32.777421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:32.777303 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" podStartSLOduration=1.036351919 podStartE2EDuration="1.777287009s" totalImagesPullingTime="740.93509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:20:32.776764658 +0000 UTC m=+1914.627524490" watchObservedRunningTime="2026-07-21 22:20:32.777287009 +0000 UTC m=+1914.628046841" Jul 21 22:20:35.340368 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:35.340339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:36.005325 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:36.005297 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:36.101451 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:36.101419 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:36.487510 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:36.487481 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:38.767984 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:38.767950 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:42.377088 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.377053 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:42.480079 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.480045 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:42.743339 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.743271 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq"] Jul 21 22:20:42.746524 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.746506 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:42.749555 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.749536 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:20:42.754368 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.754347 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq"] Jul 21 22:20:42.772717 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.772693 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq\" (UID: \"d27ae2e1-571c-4a29-967c-db33ed05aada\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:42.873648 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.873599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq\" (UID: \"d27ae2e1-571c-4a29-967c-db33ed05aada\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:42.874135 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:42.874116 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq\" (UID: \"d27ae2e1-571c-4a29-967c-db33ed05aada\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:43.057209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:43.057139 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:43.186554 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:43.186529 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq"] Jul 21 22:20:43.188753 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:20:43.188723 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27ae2e1_571c_4a29_967c_db33ed05aada.slice/crio-cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc WatchSource:0}: Error finding container cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc: Status 404 returned error can't find the container with id cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc Jul 21 22:20:43.807156 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:43.807076 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" event={"ID":"d27ae2e1-571c-4a29-967c-db33ed05aada","Type":"ContainerStarted","Data":"998ff170c4cddc5a362effcb72efcacbbadb9c451c819971cea7837d8f11b1c9"} Jul 21 22:20:43.807156 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:43.807139 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" event={"ID":"d27ae2e1-571c-4a29-967c-db33ed05aada","Type":"ContainerStarted","Data":"cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc"} Jul 21 22:20:43.829355 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:43.829312 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" podStartSLOduration=1.8292955260000001 podStartE2EDuration="1.829295526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:20:43.82746945 +0000 UTC m=+1925.678229282" watchObservedRunningTime="2026-07-21 22:20:43.829295526 +0000 UTC m=+1925.680055356" Jul 21 22:20:44.812731 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:44.812692 2555 generic.go:356] "Generic (PLEG): container finished" podID="d27ae2e1-571c-4a29-967c-db33ed05aada" containerID="998ff170c4cddc5a362effcb72efcacbbadb9c451c819971cea7837d8f11b1c9" exitCode=0 Jul 21 22:20:44.813129 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:44.812779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" event={"ID":"d27ae2e1-571c-4a29-967c-db33ed05aada","Type":"ContainerDied","Data":"998ff170c4cddc5a362effcb72efcacbbadb9c451c819971cea7837d8f11b1c9"} Jul 21 22:20:45.948713 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:45.948691 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:45.996565 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:45.996541 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca\") pod \"d27ae2e1-571c-4a29-967c-db33ed05aada\" (UID: \"d27ae2e1-571c-4a29-967c-db33ed05aada\") " Jul 21 22:20:45.996859 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:45.996840 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d27ae2e1-571c-4a29-967c-db33ed05aada" (UID: "d27ae2e1-571c-4a29-967c-db33ed05aada"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:20:46.098032 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:46.098010 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27ae2e1-571c-4a29-967c-db33ed05aada-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:20:46.823016 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:46.822988 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" Jul 21 22:20:46.823168 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:46.823014 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq" event={"ID":"d27ae2e1-571c-4a29-967c-db33ed05aada","Type":"ContainerDied","Data":"cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc"} Jul 21 22:20:46.823168 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:46.823061 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cead4f8cb68b1905d3427ed4a2acb9aef72c9e017a5e2030971ffadd5fcfe6dc" Jul 21 22:20:53.276514 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.276476 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:53.277147 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.276768 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" podUID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" containerName="maas-api" containerID="cri-o://62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff" gracePeriod=30 Jul 21 22:20:53.309620 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.309577 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:53.530011 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.529949 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:53.650662 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.650631 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jmqv\" (UniqueName: \"kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv\") pod \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " Jul 21 22:20:53.650806 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.650692 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca\") pod \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " Jul 21 22:20:53.650806 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.650734 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls\") pod \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\" (UID: \"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89\") " Jul 21 22:20:53.651090 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.651055 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" (UID: "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:20:53.652755 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.652726 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" (UID: "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:20:53.652867 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.652775 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv" (OuterVolumeSpecName: "kube-api-access-2jmqv") pod "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" (UID: "5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89"). InnerVolumeSpecName "kube-api-access-2jmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:20:53.751899 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.751866 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jmqv\" (UniqueName: \"kubernetes.io/projected/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-kube-api-access-2jmqv\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:20:53.751899 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.751897 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:20:53.751899 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.751906 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:20:53.854903 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.854878 2555 generic.go:356] "Generic (PLEG): container finished" podID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" containerID="62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff" exitCode=0 Jul 21 22:20:53.854996 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.854925 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" Jul 21 22:20:53.854996 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.854963 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" event={"ID":"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89","Type":"ContainerDied","Data":"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff"} Jul 21 22:20:53.855065 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.855020 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w" event={"ID":"5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89","Type":"ContainerDied","Data":"c4cf3c62bbdb0cfadccd67ce51dcc37ad4443c9838f3dc9675f24cc0c09f462d"} Jul 21 22:20:53.855065 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.855043 2555 scope.go:117] "RemoveContainer" containerID="62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff" Jul 21 22:20:53.866677 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.866659 2555 scope.go:117] "RemoveContainer" containerID="62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff" Jul 21 22:20:53.866946 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:20:53.866928 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff\": container with ID starting with 62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff not found: ID does not exist" containerID="62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff" Jul 21 22:20:53.867008 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.866962 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff"} err="failed to get container status \"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff\": rpc error: code = NotFound desc = could not find container \"62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff\": container with ID starting with 62204ee3f04fc78c3eda38abc7d7d6b5bbd87f7a6716232f9be751ac9c1650ff not found: ID does not exist" Jul 21 22:20:53.879705 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.879684 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:53.884100 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:53.884079 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e0aec917-dd78b46b5-sks5w"] Jul 21 22:20:54.664703 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:54.664667 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" path="/var/lib/kubelet/pods/5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89/volumes" Jul 21 22:20:57.824040 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:57.824004 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:57.825163 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:57.825135 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq"] Jul 21 22:20:57.827103 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:57.827080 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0aec917-6pcxq"] Jul 21 22:20:58.224432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:58.224343 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:20:58.663730 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:20:58.663702 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27ae2e1-571c-4a29-967c-db33ed05aada" path="/var/lib/kubelet/pods/d27ae2e1-571c-4a29-967c-db33ed05aada/volumes" Jul 21 22:21:04.551945 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:04.551907 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:04.631763 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:04.631729 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:04.824628 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:04.824565 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:05.068651 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:05.068601 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:05.203209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:05.203131 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:05.295244 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:05.295215 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:05.450201 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:05.450166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:11.959882 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:11.959845 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:12.033549 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:12.033515 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:12.186748 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:12.186696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:12.330260 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:12.330229 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:18.245054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:18.245019 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:18.317888 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:18.317854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:25.137054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:25.137017 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:25.449195 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:25.449107 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:25.605497 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:25.605464 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.036780 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.036751 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.130045 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130000 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:21:28.130219 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130052 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.130448 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130433 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27ae2e1-571c-4a29-967c-db33ed05aada" containerName="revoke-keys" Jul 21 22:21:28.130496 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130452 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27ae2e1-571c-4a29-967c-db33ed05aada" containerName="revoke-keys" Jul 21 22:21:28.130496 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130486 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" containerName="maas-api" Jul 21 22:21:28.130496 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130495 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" containerName="maas-api" Jul 21 22:21:28.130592 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130561 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9c2dbf-5882-4bd7-a3fc-06fc9e006e89" containerName="maas-api" Jul 21 22:21:28.130592 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.130575 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27ae2e1-571c-4a29-967c-db33ed05aada" containerName="revoke-keys" Jul 21 22:21:28.133551 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.133530 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.136259 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.136245 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6699167f-serving-cert\"" Jul 21 22:21:28.136354 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.136294 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:21:28.144187 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.144152 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:21:28.210242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.210212 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.240581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.240551 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.240778 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.240595 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.240778 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.240639 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pqp\" (UniqueName: \"kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.341934 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.341895 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.342095 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.341957 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.342095 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.341988 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pqp\" (UniqueName: \"kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.342176 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:21:28.342126 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-serving-cert: secret "maas-api-e2e-mt-6699167f-serving-cert" not found Jul 21 22:21:28.342217 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:21:28.342199 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls podName:6f9858ea-5b58-4f2c-b2f9-0195004b61c2 nodeName:}" failed. No retries permitted until 2026-07-21 22:21:28.842182548 +0000 UTC m=+1970.692942357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls") pod "maas-api-e2e-mt-6699167f-7df5849974-lj4hx" (UID: "6f9858ea-5b58-4f2c-b2f9-0195004b61c2") : secret "maas-api-e2e-mt-6699167f-serving-cert" not found Jul 21 22:21:28.342503 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.342485 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.355302 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.355272 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pqp\" (UniqueName: \"kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.409025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.408986 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.720032 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.719948 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.821151 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.821116 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:28.847155 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.847122 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:28.849560 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:28.849535 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") pod \"maas-api-e2e-mt-6699167f-7df5849974-lj4hx\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:29.045102 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:29.045020 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:29.200585 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:29.200555 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:21:29.201426 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:21:29.201403 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9858ea_5b58_4f2c_b2f9_0195004b61c2.slice/crio-c110375e0fc298451ce5c06a67f3efcdd0773ba591af836e91fff456610f8d4b WatchSource:0}: Error finding container c110375e0fc298451ce5c06a67f3efcdd0773ba591af836e91fff456610f8d4b: Status 404 returned error can't find the container with id c110375e0fc298451ce5c06a67f3efcdd0773ba591af836e91fff456610f8d4b Jul 21 22:21:29.203152 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:29.203132 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 22:21:30.018983 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:30.018900 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" event={"ID":"6f9858ea-5b58-4f2c-b2f9-0195004b61c2","Type":"ContainerStarted","Data":"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77"} Jul 21 22:21:30.018983 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:30.018952 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" event={"ID":"6f9858ea-5b58-4f2c-b2f9-0195004b61c2","Type":"ContainerStarted","Data":"c110375e0fc298451ce5c06a67f3efcdd0773ba591af836e91fff456610f8d4b"} Jul 21 22:21:30.019197 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:30.019041 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:30.037828 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:30.037779 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" podStartSLOduration=1.727238064 podStartE2EDuration="2.03776605s" totalImagesPullingTime="310.527986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:21:30.036556355 +0000 UTC m=+1971.887316189" watchObservedRunningTime="2026-07-21 22:21:30.03776605 +0000 UTC m=+1971.888525881" Jul 21 22:21:32.548813 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:32.548779 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:35.159152 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:35.159121 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:35.217936 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:35.217896 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:36.031030 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:36.031004 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:21:43.147376 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:43.147339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:21:56.727023 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:21:56.726984 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:22:11.091818 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.091779 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq"] Jul 21 22:22:11.094996 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.094979 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:11.097719 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.097692 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:22:11.103314 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.103287 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq"] Jul 21 22:22:11.172818 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.172791 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6699167f-s4nwq\" (UID: \"6a266807-54ec-4023-905d-f2d05ebeea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:11.273291 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.273267 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6699167f-s4nwq\" (UID: \"6a266807-54ec-4023-905d-f2d05ebeea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:11.273842 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.273825 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6699167f-s4nwq\" (UID: \"6a266807-54ec-4023-905d-f2d05ebeea4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:11.406171 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.406121 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:11.531012 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:11.530926 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq"] Jul 21 22:22:11.533357 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:22:11.533327 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a266807_54ec_4023_905d_f2d05ebeea4d.slice/crio-b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f WatchSource:0}: Error finding container b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f: Status 404 returned error can't find the container with id b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f Jul 21 22:22:12.211079 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:12.211047 2555 generic.go:356] "Generic (PLEG): container finished" podID="6a266807-54ec-4023-905d-f2d05ebeea4d" containerID="d6b8d85ccb4aae490550dbf423f7b8245014674ec6357dea2f4f6da2278e528f" exitCode=0 Jul 21 22:22:12.211426 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:12.211128 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" event={"ID":"6a266807-54ec-4023-905d-f2d05ebeea4d","Type":"ContainerDied","Data":"d6b8d85ccb4aae490550dbf423f7b8245014674ec6357dea2f4f6da2278e528f"} Jul 21 22:22:12.211426 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:12.211174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" event={"ID":"6a266807-54ec-4023-905d-f2d05ebeea4d","Type":"ContainerStarted","Data":"b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f"} Jul 21 22:22:13.359590 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:13.359568 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:13.489296 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:13.489229 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca\") pod \"6a266807-54ec-4023-905d-f2d05ebeea4d\" (UID: \"6a266807-54ec-4023-905d-f2d05ebeea4d\") " Jul 21 22:22:13.489554 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:13.489533 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a266807-54ec-4023-905d-f2d05ebeea4d" (UID: "6a266807-54ec-4023-905d-f2d05ebeea4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:22:13.589847 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:13.589824 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a266807-54ec-4023-905d-f2d05ebeea4d-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:22:14.221926 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:14.221893 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" Jul 21 22:22:14.221926 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:14.221904 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq" event={"ID":"6a266807-54ec-4023-905d-f2d05ebeea4d","Type":"ContainerDied","Data":"b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f"} Jul 21 22:22:14.222161 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:14.221998 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b709fd14da7159a6c18c9782aa2288f04b3dc1659bf13b4e09e2480893e3f63f" Jul 21 22:22:26.609755 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.609717 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:22:26.610226 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.609975 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" podUID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" containerName="maas-api" containerID="cri-o://462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77" gracePeriod=30 Jul 21 22:22:26.635868 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.635839 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:22:26.846501 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.846474 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:22:26.998577 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.998505 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8pqp\" (UniqueName: \"kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp\") pod \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " Jul 21 22:22:26.998577 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.998536 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") pod \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " Jul 21 22:22:26.998797 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.998632 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca\") pod \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\" (UID: \"6f9858ea-5b58-4f2c-b2f9-0195004b61c2\") " Jul 21 22:22:26.999024 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:26.999000 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f9858ea-5b58-4f2c-b2f9-0195004b61c2" (UID: "6f9858ea-5b58-4f2c-b2f9-0195004b61c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:22:27.000475 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.000454 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f9858ea-5b58-4f2c-b2f9-0195004b61c2" (UID: "6f9858ea-5b58-4f2c-b2f9-0195004b61c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:22:27.000530 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.000476 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp" (OuterVolumeSpecName: "kube-api-access-r8pqp") pod "6f9858ea-5b58-4f2c-b2f9-0195004b61c2" (UID: "6f9858ea-5b58-4f2c-b2f9-0195004b61c2"). InnerVolumeSpecName "kube-api-access-r8pqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:22:27.099552 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.099522 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:22:27.099552 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.099548 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pqp\" (UniqueName: \"kubernetes.io/projected/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-kube-api-access-r8pqp\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:22:27.099724 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.099562 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f9858ea-5b58-4f2c-b2f9-0195004b61c2-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:22:27.290036 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.289950 2555 generic.go:356] "Generic (PLEG): container finished" podID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" containerID="462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77" exitCode=0 Jul 21 22:22:27.290036 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.290009 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" Jul 21 22:22:27.290242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.290034 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" event={"ID":"6f9858ea-5b58-4f2c-b2f9-0195004b61c2","Type":"ContainerDied","Data":"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77"} Jul 21 22:22:27.290242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.290110 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx" event={"ID":"6f9858ea-5b58-4f2c-b2f9-0195004b61c2","Type":"ContainerDied","Data":"c110375e0fc298451ce5c06a67f3efcdd0773ba591af836e91fff456610f8d4b"} Jul 21 22:22:27.290242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.290135 2555 scope.go:117] "RemoveContainer" containerID="462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77" Jul 21 22:22:27.299541 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.299520 2555 scope.go:117] "RemoveContainer" containerID="462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77" Jul 21 22:22:27.299927 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:22:27.299898 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77\": container with ID starting with 462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77 not found: ID does not exist" containerID="462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77" Jul 21 22:22:27.299995 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.299941 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77"} err="failed to get container status \"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77\": rpc error: code = NotFound desc = could not find container \"462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77\": container with ID starting with 462b47c37fe6d8d0c31507ba700b67199b20055e45ba4b2daae47f8f5cf0dd77 not found: ID does not exist" Jul 21 22:22:27.311241 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.311220 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:22:27.314964 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:27.314942 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6699167f-7df5849974-lj4hx"] Jul 21 22:22:28.664343 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:28.664316 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" path="/var/lib/kubelet/pods/6f9858ea-5b58-4f2c-b2f9-0195004b61c2/volumes" Jul 21 22:22:31.187072 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:31.187038 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:22:31.187451 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:31.187121 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq"] Jul 21 22:22:31.190387 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:31.190364 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6699167f-s4nwq"] Jul 21 22:22:32.664019 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:32.663986 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a266807-54ec-4023-905d-f2d05ebeea4d" path="/var/lib/kubelet/pods/6a266807-54ec-4023-905d-f2d05ebeea4d/volumes" Jul 21 22:22:33.619076 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:33.619033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:22:38.661065 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:38.661029 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:22:38.710893 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:38.710866 2555 scope.go:117] "RemoveContainer" containerID="aa1e20609dab0d4d1f1b6653675852cd5e194684f657db22fa1a209d5c0e633f" Jul 21 22:22:48.917638 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:22:48.917588 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:05.378383 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:05.378302 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:15.539502 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:15.539466 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:16.023154 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:16.023111 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:21.379073 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:21.379035 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:21.480959 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:21.480922 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:22.537377 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:22.537328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:22.602118 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:22.602090 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:22.680863 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:22.680834 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:23.766953 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:23.766907 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:23.867297 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:23.867264 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:23.969581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:23.969548 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:24.733110 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:24.733081 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:24.839781 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:24.839750 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:24.930580 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:24.930550 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:26.312869 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:26.312821 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:32.678987 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:32.678942 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:32.780760 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:32.780734 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:33.382321 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:33.382287 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:38.733298 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:38.733271 2555 scope.go:117] "RemoveContainer" containerID="3641f5ad9747e24d2704f61e19b0b0b0f2774501ee23843dd41cab83457387f2" Jul 21 22:23:39.716970 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:39.716918 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:39.824124 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:39.824095 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:39.922029 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:39.921998 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:50.481113 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:50.481070 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:50.868992 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:50.868957 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:23:53.440251 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:23:53.440211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:11.832666 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:11.832625 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:16.570261 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:16.570224 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:16.670961 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:16.670930 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:18.171178 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:18.171142 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:18.271576 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:18.271529 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:18.538593 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:18.538515 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:18.863148 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:18.863111 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:19.070198 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:19.070166 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:21.915829 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:21.915794 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.016535 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016504 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:24:22.016900 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016886 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" containerName="maas-api" Jul 21 22:24:22.016978 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016902 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" containerName="maas-api" Jul 21 22:24:22.016978 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016927 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a266807-54ec-4023-905d-f2d05ebeea4d" containerName="revoke-keys" Jul 21 22:24:22.016978 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016934 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a266807-54ec-4023-905d-f2d05ebeea4d" containerName="revoke-keys" Jul 21 22:24:22.017130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016982 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a266807-54ec-4023-905d-f2d05ebeea4d" containerName="revoke-keys" Jul 21 22:24:22.017130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.016990 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f9858ea-5b58-4f2c-b2f9-0195004b61c2" containerName="maas-api" Jul 21 22:24:22.019737 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.019719 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.021562 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.021534 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.022909 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.022886 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3f5ae6-serving-cert\"" Jul 21 22:24:22.023024 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.022989 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:24:22.033308 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.033287 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:24:22.061297 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.061270 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.061424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.061315 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t7r\" (UniqueName: \"kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.061424 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.061378 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.081582 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.081558 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.162667 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.162599 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t7r\" (UniqueName: \"kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.162856 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.162698 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.162856 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.162751 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.162973 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:24:22.162875 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-serving-cert: secret "maas-api-e2e-inf-a-3f5ae6-serving-cert" not found Jul 21 22:24:22.162973 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:24:22.162949 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls podName:a7b60e60-ff49-4a1c-adae-aa169a4f6846 nodeName:}" failed. No retries permitted until 2026-07-21 22:24:22.662929197 +0000 UTC m=+2144.513689013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls") pod "maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" (UID: "a7b60e60-ff49-4a1c-adae-aa169a4f6846") : secret "maas-api-e2e-inf-a-3f5ae6-serving-cert" not found Jul 21 22:24:22.163302 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.163283 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.173992 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.173930 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t7r\" (UniqueName: \"kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.457178 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.457094 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.644147 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.644115 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.665858 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.665826 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.668263 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.668243 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") pod \"maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:22.744088 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.744027 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:22.929827 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:22.929796 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:23.061231 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:23.061206 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:24:23.063301 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:24:23.063267 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b60e60_ff49_4a1c_adae_aa169a4f6846.slice/crio-048d6a116bef75e80a56c742dcc3e5f18c731a133d02eb0dfff2f3507e1ad4c7 WatchSource:0}: Error finding container 048d6a116bef75e80a56c742dcc3e5f18c731a133d02eb0dfff2f3507e1ad4c7: Status 404 returned error can't find the container with id 048d6a116bef75e80a56c742dcc3e5f18c731a133d02eb0dfff2f3507e1ad4c7 Jul 21 22:24:23.823156 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:23.823125 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" event={"ID":"a7b60e60-ff49-4a1c-adae-aa169a4f6846","Type":"ContainerStarted","Data":"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9"} Jul 21 22:24:23.823398 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:23.823174 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" event={"ID":"a7b60e60-ff49-4a1c-adae-aa169a4f6846","Type":"ContainerStarted","Data":"048d6a116bef75e80a56c742dcc3e5f18c731a133d02eb0dfff2f3507e1ad4c7"} Jul 21 22:24:23.823398 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:23.823258 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:23.845784 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:23.845733 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" podStartSLOduration=2.302834253 podStartE2EDuration="2.845718533s" totalImagesPullingTime="542.88428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:24:23.843287849 +0000 UTC m=+2145.694047680" watchObservedRunningTime="2026-07-21 22:24:23.845718533 +0000 UTC m=+2145.696478364" Jul 21 22:24:26.433203 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:26.433150 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:26.790744 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:26.790661 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:27.155841 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:27.155804 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:27.867237 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:27.867194 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:29.833300 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:29.833233 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:24:30.059076 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.059049 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:30.143129 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.143054 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:30.146523 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.146499 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:24:30.150120 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.150101 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.152580 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.152564 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4ad703-serving-cert\"" Jul 21 22:24:30.161627 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.161582 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:24:30.206837 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.206812 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:30.226985 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.226960 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.227082 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.227003 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.227082 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.227054 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99m8\" (UniqueName: \"kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.328267 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.328234 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.328409 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.328290 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.328409 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.328338 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99m8\" (UniqueName: \"kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.329023 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.329004 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.330848 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.330824 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.339689 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.339668 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99m8\" (UniqueName: \"kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8\") pod \"maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.408841 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.408788 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:30.460955 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.460923 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:30.591846 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.591820 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:24:30.593739 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:24:30.593713 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda624852e_925a_4d48_a196_8b5b2f3b4a93.slice/crio-97fb8ecee139c8a1088448fa6bb94dc8f770b0433c87777119441ca371331482 WatchSource:0}: Error finding container 97fb8ecee139c8a1088448fa6bb94dc8f770b0433c87777119441ca371331482: Status 404 returned error can't find the container with id 97fb8ecee139c8a1088448fa6bb94dc8f770b0433c87777119441ca371331482 Jul 21 22:24:30.766584 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.766497 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:30.855414 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.855329 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" event={"ID":"a624852e-925a-4d48-a196-8b5b2f3b4a93","Type":"ContainerStarted","Data":"97fb8ecee139c8a1088448fa6bb94dc8f770b0433c87777119441ca371331482"} Jul 21 22:24:30.874438 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:30.869211 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:31.861188 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:31.861153 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" event={"ID":"a624852e-925a-4d48-a196-8b5b2f3b4a93","Type":"ContainerStarted","Data":"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122"} Jul 21 22:24:31.861661 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:31.861283 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:31.880799 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:31.880751 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" podStartSLOduration=1.5359008570000001 podStartE2EDuration="1.880734795s" totalImagesPullingTime="344.833938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:24:31.879470662 +0000 UTC m=+2153.730230492" watchObservedRunningTime="2026-07-21 22:24:31.880734795 +0000 UTC m=+2153.731494625" Jul 21 22:24:34.456246 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:34.456205 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:37.870717 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:37.870692 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:24:38.751835 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:38.751806 2555 scope.go:117] "RemoveContainer" containerID="b223fa255c2f44a381c78075020c4805c901dd15eaf5101400b0ae24d1952aa4" Jul 21 22:24:38.759384 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:38.759366 2555 scope.go:117] "RemoveContainer" containerID="fb5800511735b660625bd77a2ba45a85119fcf618d7c25988ac1eeaac83cad65" Jul 21 22:24:43.663455 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:43.663424 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:43.755441 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:43.755409 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:52.924657 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:52.924623 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:24:53.025594 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:24:53.025559 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:25:00.597130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:25:00.597100 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:25:35.617505 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:25:35.617469 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:20.141021 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:20.140985 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:21.081014 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.080977 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:21.198435 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.198406 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq"] Jul 21 22:26:21.201740 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.201723 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:21.204378 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.204362 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:26:21.209772 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.209745 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq"] Jul 21 22:26:21.344309 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.344242 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq\" (UID: \"6ef63bb9-9b17-4369-b909-fa382b62dd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:21.445022 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.444996 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq\" (UID: \"6ef63bb9-9b17-4369-b909-fa382b62dd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:21.445552 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.445534 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq\" (UID: \"6ef63bb9-9b17-4369-b909-fa382b62dd3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:21.512235 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.512213 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:21.641290 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.641262 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq"] Jul 21 22:26:21.642924 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:26:21.642893 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef63bb9_9b17_4369_b909_fa382b62dd3e.slice/crio-6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9 WatchSource:0}: Error finding container 6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9: Status 404 returned error can't find the container with id 6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9 Jul 21 22:26:21.739721 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.739682 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:21.840667 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:21.840641 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:22.353480 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:22.353449 2555 generic.go:356] "Generic (PLEG): container finished" podID="6ef63bb9-9b17-4369-b909-fa382b62dd3e" containerID="6dc045a28d2725937fc57db7ce672933e6881e3f12cfdcd623e9adca7a25ba71" exitCode=0 Jul 21 22:26:22.353830 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:22.353497 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" event={"ID":"6ef63bb9-9b17-4369-b909-fa382b62dd3e","Type":"ContainerDied","Data":"6dc045a28d2725937fc57db7ce672933e6881e3f12cfdcd623e9adca7a25ba71"} Jul 21 22:26:22.353830 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:22.353541 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" event={"ID":"6ef63bb9-9b17-4369-b909-fa382b62dd3e","Type":"ContainerStarted","Data":"6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9"} Jul 21 22:26:23.485674 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:23.485653 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:23.559425 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:23.559398 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca\") pod \"6ef63bb9-9b17-4369-b909-fa382b62dd3e\" (UID: \"6ef63bb9-9b17-4369-b909-fa382b62dd3e\") " Jul 21 22:26:23.559761 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:23.559739 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ef63bb9-9b17-4369-b909-fa382b62dd3e" (UID: "6ef63bb9-9b17-4369-b909-fa382b62dd3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:26:23.660748 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:23.660697 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef63bb9-9b17-4369-b909-fa382b62dd3e-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:24.364321 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:24.364283 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" event={"ID":"6ef63bb9-9b17-4369-b909-fa382b62dd3e","Type":"ContainerDied","Data":"6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9"} Jul 21 22:26:24.364321 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:24.364312 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq" Jul 21 22:26:24.364321 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:24.364327 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6177997dee4a015ff79cebf47290d44b4bc80d35ba9917dc857d159caeac5de9" Jul 21 22:26:31.645890 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:31.645847 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:26:31.646339 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:31.646094 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" podUID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" containerName="maas-api" containerID="cri-o://8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9" gracePeriod=30 Jul 21 22:26:31.672487 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:31.672460 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:31.894494 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:31.894467 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:26:32.028558 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.028487 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca\") pod \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " Jul 21 22:26:32.028735 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.028592 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") pod \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " Jul 21 22:26:32.028735 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.028654 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6t7r\" (UniqueName: \"kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r\") pod \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\" (UID: \"a7b60e60-ff49-4a1c-adae-aa169a4f6846\") " Jul 21 22:26:32.028938 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.028901 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7b60e60-ff49-4a1c-adae-aa169a4f6846" (UID: "a7b60e60-ff49-4a1c-adae-aa169a4f6846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:26:32.030653 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.030629 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r" (OuterVolumeSpecName: "kube-api-access-s6t7r") pod "a7b60e60-ff49-4a1c-adae-aa169a4f6846" (UID: "a7b60e60-ff49-4a1c-adae-aa169a4f6846"). InnerVolumeSpecName "kube-api-access-s6t7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:32.030727 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.030694 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7b60e60-ff49-4a1c-adae-aa169a4f6846" (UID: "a7b60e60-ff49-4a1c-adae-aa169a4f6846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:26:32.129685 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.129662 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b60e60-ff49-4a1c-adae-aa169a4f6846-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:32.129685 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.129683 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6t7r\" (UniqueName: \"kubernetes.io/projected/a7b60e60-ff49-4a1c-adae-aa169a4f6846-kube-api-access-s6t7r\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:32.129816 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.129692 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b60e60-ff49-4a1c-adae-aa169a4f6846-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:32.400743 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.400707 2555 generic.go:356] "Generic (PLEG): container finished" podID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" containerID="8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9" exitCode=0 Jul 21 22:26:32.400868 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.400748 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" Jul 21 22:26:32.400868 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.400796 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" event={"ID":"a7b60e60-ff49-4a1c-adae-aa169a4f6846","Type":"ContainerDied","Data":"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9"} Jul 21 22:26:32.400958 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.400882 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k" event={"ID":"a7b60e60-ff49-4a1c-adae-aa169a4f6846","Type":"ContainerDied","Data":"048d6a116bef75e80a56c742dcc3e5f18c731a133d02eb0dfff2f3507e1ad4c7"} Jul 21 22:26:32.400958 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.400917 2555 scope.go:117] "RemoveContainer" containerID="8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9" Jul 21 22:26:32.409445 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.409429 2555 scope.go:117] "RemoveContainer" containerID="8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9" Jul 21 22:26:32.409739 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:26:32.409718 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9\": container with ID starting with 8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9 not found: ID does not exist" containerID="8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9" Jul 21 22:26:32.409797 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.409747 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9"} err="failed to get container status \"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9\": rpc error: code = NotFound desc = could not find container \"8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9\": container with ID starting with 8dfa0a9ee52bae5f6d06e91881d3971592fa6c8627bf57a17648fef4da5098a9 not found: ID does not exist" Jul 21 22:26:32.423945 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.423922 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:26:32.428083 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.428063 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3f5ae6-79d6bd7c74-jst6k"] Jul 21 22:26:32.664219 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:32.664157 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" path="/var/lib/kubelet/pods/a7b60e60-ff49-4a1c-adae-aa169a4f6846/volumes" Jul 21 22:26:36.265407 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:36.265376 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:36.278592 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:36.278562 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq"] Jul 21 22:26:36.281778 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:36.281753 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3f5ae6-6p4lq"] Jul 21 22:26:36.667280 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:36.667244 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:36.667536 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:36.667517 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef63bb9-9b17-4369-b909-fa382b62dd3e" path="/var/lib/kubelet/pods/6ef63bb9-9b17-4369-b909-fa382b62dd3e/volumes" Jul 21 22:26:42.658934 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:42.658895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:42.758832 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:42.758800 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:42.925591 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:42.925516 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:43.172115 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.172081 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:43.285678 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.285597 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5"] Jul 21 22:26:43.285953 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.285939 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" containerName="maas-api" Jul 21 22:26:43.285993 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.285954 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" containerName="maas-api" Jul 21 22:26:43.285993 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.285964 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef63bb9-9b17-4369-b909-fa382b62dd3e" containerName="revoke-keys" Jul 21 22:26:43.285993 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.285969 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef63bb9-9b17-4369-b909-fa382b62dd3e" containerName="revoke-keys" Jul 21 22:26:43.286097 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.286022 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b60e60-ff49-4a1c-adae-aa169a4f6846" containerName="maas-api" Jul 21 22:26:43.286097 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.286033 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef63bb9-9b17-4369-b909-fa382b62dd3e" containerName="revoke-keys" Jul 21 22:26:43.288903 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.288878 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:43.291794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.291776 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:26:43.297028 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.297006 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5"] Jul 21 22:26:43.421196 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.421169 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ad703-857f5\" (UID: \"33b37c12-8ede-4011-be28-c4af6d630914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:43.521764 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.521732 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ad703-857f5\" (UID: \"33b37c12-8ede-4011-be28-c4af6d630914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:43.522318 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.522298 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4ad703-857f5\" (UID: \"33b37c12-8ede-4011-be28-c4af6d630914\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:43.599868 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.599836 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:43.729590 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.729566 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5"] Jul 21 22:26:43.732222 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:26:43.732197 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b37c12_8ede_4011_be28_c4af6d630914.slice/crio-e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732 WatchSource:0}: Error finding container e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732: Status 404 returned error can't find the container with id e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732 Jul 21 22:26:43.823114 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.823086 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:43.928358 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:43.928292 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:44.458585 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:44.458557 2555 generic.go:356] "Generic (PLEG): container finished" podID="33b37c12-8ede-4011-be28-c4af6d630914" containerID="89f3af106926027b3ba48ff9f136a732387cef73d07ee98342747bee033db0ea" exitCode=0 Jul 21 22:26:44.458708 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:44.458648 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" event={"ID":"33b37c12-8ede-4011-be28-c4af6d630914","Type":"ContainerDied","Data":"89f3af106926027b3ba48ff9f136a732387cef73d07ee98342747bee033db0ea"} Jul 21 22:26:44.458708 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:44.458703 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" event={"ID":"33b37c12-8ede-4011-be28-c4af6d630914","Type":"ContainerStarted","Data":"e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732"} Jul 21 22:26:45.587727 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:45.587703 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:45.740624 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:45.740527 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca\") pod \"33b37c12-8ede-4011-be28-c4af6d630914\" (UID: \"33b37c12-8ede-4011-be28-c4af6d630914\") " Jul 21 22:26:45.740881 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:45.740860 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33b37c12-8ede-4011-be28-c4af6d630914" (UID: "33b37c12-8ede-4011-be28-c4af6d630914"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:26:45.841047 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:45.841022 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b37c12-8ede-4011-be28-c4af6d630914-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:46.468235 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:46.468202 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" event={"ID":"33b37c12-8ede-4011-be28-c4af6d630914","Type":"ContainerDied","Data":"e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732"} Jul 21 22:26:46.468401 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:46.468255 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e31b9f9bceea214cf92828a90d2b31c4f84ff63d761dd7ab4830b80c10279732" Jul 21 22:26:46.468401 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:46.468214 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5" Jul 21 22:26:53.758432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:53.758402 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:26:53.758973 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:53.758599 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" podUID="a624852e-925a-4d48-a196-8b5b2f3b4a93" containerName="maas-api" containerID="cri-o://de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122" gracePeriod=30 Jul 21 22:26:53.858395 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:53.858365 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:54.009199 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.009145 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:26:54.102552 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.102526 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p99m8\" (UniqueName: \"kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8\") pod \"a624852e-925a-4d48-a196-8b5b2f3b4a93\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " Jul 21 22:26:54.102740 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.102589 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls\") pod \"a624852e-925a-4d48-a196-8b5b2f3b4a93\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " Jul 21 22:26:54.102740 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.102643 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca\") pod \"a624852e-925a-4d48-a196-8b5b2f3b4a93\" (UID: \"a624852e-925a-4d48-a196-8b5b2f3b4a93\") " Jul 21 22:26:54.102979 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.102955 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a624852e-925a-4d48-a196-8b5b2f3b4a93" (UID: "a624852e-925a-4d48-a196-8b5b2f3b4a93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:26:54.104590 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.104567 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a624852e-925a-4d48-a196-8b5b2f3b4a93" (UID: "a624852e-925a-4d48-a196-8b5b2f3b4a93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:26:54.104707 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.104662 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8" (OuterVolumeSpecName: "kube-api-access-p99m8") pod "a624852e-925a-4d48-a196-8b5b2f3b4a93" (UID: "a624852e-925a-4d48-a196-8b5b2f3b4a93"). InnerVolumeSpecName "kube-api-access-p99m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:54.203946 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.203924 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a624852e-925a-4d48-a196-8b5b2f3b4a93-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:54.203946 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.203945 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a624852e-925a-4d48-a196-8b5b2f3b4a93-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:54.204073 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.203956 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99m8\" (UniqueName: \"kubernetes.io/projected/a624852e-925a-4d48-a196-8b5b2f3b4a93-kube-api-access-p99m8\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:26:54.505195 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.505160 2555 generic.go:356] "Generic (PLEG): container finished" podID="a624852e-925a-4d48-a196-8b5b2f3b4a93" containerID="de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122" exitCode=0 Jul 21 22:26:54.505313 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.505209 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" Jul 21 22:26:54.505313 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.505210 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" event={"ID":"a624852e-925a-4d48-a196-8b5b2f3b4a93","Type":"ContainerDied","Data":"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122"} Jul 21 22:26:54.505313 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.505306 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l" event={"ID":"a624852e-925a-4d48-a196-8b5b2f3b4a93","Type":"ContainerDied","Data":"97fb8ecee139c8a1088448fa6bb94dc8f770b0433c87777119441ca371331482"} Jul 21 22:26:54.505432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.505334 2555 scope.go:117] "RemoveContainer" containerID="de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122" Jul 21 22:26:54.514439 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.514423 2555 scope.go:117] "RemoveContainer" containerID="de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122" Jul 21 22:26:54.514713 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:26:54.514689 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122\": container with ID starting with de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122 not found: ID does not exist" containerID="de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122" Jul 21 22:26:54.514786 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.514726 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122"} err="failed to get container status \"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122\": rpc error: code = NotFound desc = could not find container \"de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122\": container with ID starting with de7b73510072848fcd415f6b984e4a9a5b4b7a1d6be0d782539a8699396ea122 not found: ID does not exist" Jul 21 22:26:54.526707 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.526681 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:26:54.531069 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.531048 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ad703-b7b846d6-mtg7l"] Jul 21 22:26:54.664954 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:54.664922 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a624852e-925a-4d48-a196-8b5b2f3b4a93" path="/var/lib/kubelet/pods/a624852e-925a-4d48-a196-8b5b2f3b4a93/volumes" Jul 21 22:26:58.361115 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:58.361078 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:26:58.365443 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:58.365419 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5"] Jul 21 22:26:58.368052 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:58.368026 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4ad703-857f5"] Jul 21 22:26:58.664194 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:58.664116 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b37c12-8ede-4011-be28-c4af6d630914" path="/var/lib/kubelet/pods/33b37c12-8ede-4011-be28-c4af6d630914/volumes" Jul 21 22:26:58.756968 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:26:58.756941 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:04.650999 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:04.650960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:04.717561 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:04.717533 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:04.907994 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:04.907908 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:05.136299 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:05.136266 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:05.495581 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:05.495537 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:05.823385 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:05.823339 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:06.019556 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:06.019525 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:08.852897 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.852854 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:08.943948 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.943920 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:08.948181 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948157 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:08.948452 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948441 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a624852e-925a-4d48-a196-8b5b2f3b4a93" containerName="maas-api" Jul 21 22:27:08.948494 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948454 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a624852e-925a-4d48-a196-8b5b2f3b4a93" containerName="maas-api" Jul 21 22:27:08.948494 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948474 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b37c12-8ede-4011-be28-c4af6d630914" containerName="revoke-keys" Jul 21 22:27:08.948494 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948479 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b37c12-8ede-4011-be28-c4af6d630914" containerName="revoke-keys" Jul 21 22:27:08.948586 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948522 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a624852e-925a-4d48-a196-8b5b2f3b4a93" containerName="maas-api" Jul 21 22:27:08.948586 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.948532 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b37c12-8ede-4011-be28-c4af6d630914" containerName="revoke-keys" Jul 21 22:27:08.952752 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.952735 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:08.955271 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.955254 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:27:08.955725 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.955711 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-09bf16-serving-cert\"" Jul 21 22:27:08.961048 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:08.961023 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:09.014014 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.013991 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgmx\" (UniqueName: \"kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.014153 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.014030 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.014211 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.014144 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.034659 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.034632 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:09.114782 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.114715 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.114782 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.114757 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgmx\" (UniqueName: \"kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.114940 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.114790 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.115413 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.115395 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.117287 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.117255 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.123682 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.123661 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgmx\" (UniqueName: \"kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx\") pod \"maas-api-e2e-api-a-09bf16-6f84775479-zl5pb\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.166600 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.166571 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:09.263601 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.263578 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:09.390562 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.390534 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:09.393080 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:27:09.393052 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc729e48e_79fd_49f0_9c2e_6f7d92c53bf1.slice/crio-69d5f9a7850119d60a2e46ce420aae28325f0a60c6ef1b174081ec8bbb75756d WatchSource:0}: Error finding container 69d5f9a7850119d60a2e46ce420aae28325f0a60c6ef1b174081ec8bbb75756d: Status 404 returned error can't find the container with id 69d5f9a7850119d60a2e46ce420aae28325f0a60c6ef1b174081ec8bbb75756d Jul 21 22:27:09.394376 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.394358 2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 22:27:09.575638 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.575592 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" event={"ID":"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1","Type":"ContainerStarted","Data":"69d5f9a7850119d60a2e46ce420aae28325f0a60c6ef1b174081ec8bbb75756d"} Jul 21 22:27:09.617701 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.617675 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:09.718654 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:09.718563 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:10.581589 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:10.581551 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" event={"ID":"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1","Type":"ContainerStarted","Data":"3ba029b0f6d512b55a725457276131694def1314388abe121030e6e3fe59b35f"} Jul 21 22:27:10.581949 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:10.581710 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:10.600052 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:10.600003 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" podStartSLOduration=2.071182724 podStartE2EDuration="2.599988086s" totalImagesPullingTime="528.805362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:27:10.598303321 +0000 UTC m=+2312.449063151" watchObservedRunningTime="2026-07-21 22:27:10.599988086 +0000 UTC m=+2312.450747937" Jul 21 22:27:13.353081 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:13.353037 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:13.825192 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:13.825164 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:14.323054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:14.322990 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:14.747668 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:14.747574 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:16.592142 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:16.592110 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:16.966271 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:16.966180 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.069405 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.069375 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.083199 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.083172 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:27:17.086480 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.086461 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.088829 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.088808 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-76e244-serving-cert\"" Jul 21 22:27:17.103626 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.103583 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:27:17.166368 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.166342 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.168394 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.168375 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.168461 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.168410 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnkzw\" (UniqueName: \"kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.168503 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.168468 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.269209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.269146 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.269209 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.269191 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnkzw\" (UniqueName: \"kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.269361 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.269229 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.269361 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:27:17.269344 2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-serving-cert: secret "maas-api-e2e-api-b-76e244-serving-cert" not found Jul 21 22:27:17.269437 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:27:17.269402 2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls podName:12a9d170-03ae-4f34-ade4-2f46f50efac5 nodeName:}" failed. No retries permitted until 2026-07-21 22:27:17.769383394 +0000 UTC m=+2319.620143217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls") pod "maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" (UID: "12a9d170-03ae-4f34-ade4-2f46f50efac5") : secret "maas-api-e2e-api-b-76e244-serving-cert" not found Jul 21 22:27:17.269914 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.269893 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.285176 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.285151 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnkzw\" (UniqueName: \"kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.607905 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.607878 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.773556 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.773527 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.775873 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.775853 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") pod \"maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:17.779661 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.779637 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.882434 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.882374 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:17.996893 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:17.996862 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:18.122678 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:18.122657 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:27:18.125263 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:27:18.125230 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a9d170_03ae_4f34_ade4_2f46f50efac5.slice/crio-860601eef1523ae4bb06ea8f48a90cb1185af2467d1599ec9947ef441ce30957 WatchSource:0}: Error finding container 860601eef1523ae4bb06ea8f48a90cb1185af2467d1599ec9947ef441ce30957: Status 404 returned error can't find the container with id 860601eef1523ae4bb06ea8f48a90cb1185af2467d1599ec9947ef441ce30957 Jul 21 22:27:18.626292 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:18.626253 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" event={"ID":"12a9d170-03ae-4f34-ade4-2f46f50efac5","Type":"ContainerStarted","Data":"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba"} Jul 21 22:27:18.626747 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:18.626359 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:18.626747 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:18.626378 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" event={"ID":"12a9d170-03ae-4f34-ade4-2f46f50efac5","Type":"ContainerStarted","Data":"860601eef1523ae4bb06ea8f48a90cb1185af2467d1599ec9947ef441ce30957"} Jul 21 22:27:18.648991 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:18.648944 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" podStartSLOduration=1.26626509 podStartE2EDuration="1.648930987s" totalImagesPullingTime="382.665897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:27:18.646991886 +0000 UTC m=+2320.497751716" watchObservedRunningTime="2026-07-21 22:27:18.648930987 +0000 UTC m=+2320.499690817" Jul 21 22:27:21.389954 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:21.389914 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:24.636895 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:24.636860 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:27:29.442906 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.442875 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w"] Jul 21 22:27:29.446394 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.446376 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:29.449084 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.449062 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:27:29.454378 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.454356 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w"] Jul 21 22:27:29.554681 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.554652 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w\" (UID: \"8b5c2eeb-e754-47b7-a585-59e7c07a97f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:29.654966 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.654941 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w\" (UID: \"8b5c2eeb-e754-47b7-a585-59e7c07a97f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:29.655528 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.655504 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w\" (UID: \"8b5c2eeb-e754-47b7-a585-59e7c07a97f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:29.757468 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.757413 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:29.886390 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:29.886367 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w"] Jul 21 22:27:29.888602 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:27:29.888578 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5c2eeb_e754_47b7_a585_59e7c07a97f5.slice/crio-94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339 WatchSource:0}: Error finding container 94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339: Status 404 returned error can't find the container with id 94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339 Jul 21 22:27:30.070793 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:30.070764 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:30.170935 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:30.170905 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:30.683933 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:30.683904 2555 generic.go:356] "Generic (PLEG): container finished" podID="8b5c2eeb-e754-47b7-a585-59e7c07a97f5" containerID="c0e911793c15992b61c7d22653ee6b0d8f204d7fc189184bd2af1fd73977bd54" exitCode=0 Jul 21 22:27:30.684259 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:30.683938 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" event={"ID":"8b5c2eeb-e754-47b7-a585-59e7c07a97f5","Type":"ContainerDied","Data":"c0e911793c15992b61c7d22653ee6b0d8f204d7fc189184bd2af1fd73977bd54"} Jul 21 22:27:30.684259 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:30.683990 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" event={"ID":"8b5c2eeb-e754-47b7-a585-59e7c07a97f5","Type":"ContainerStarted","Data":"94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339"} Jul 21 22:27:31.815658 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:31.815636 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:31.871865 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:31.871842 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca\") pod \"8b5c2eeb-e754-47b7-a585-59e7c07a97f5\" (UID: \"8b5c2eeb-e754-47b7-a585-59e7c07a97f5\") " Jul 21 22:27:31.872146 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:31.872127 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b5c2eeb-e754-47b7-a585-59e7c07a97f5" (UID: "8b5c2eeb-e754-47b7-a585-59e7c07a97f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:27:31.973147 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:31.973094 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b5c2eeb-e754-47b7-a585-59e7c07a97f5-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:27:32.694046 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:32.694016 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" event={"ID":"8b5c2eeb-e754-47b7-a585-59e7c07a97f5","Type":"ContainerDied","Data":"94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339"} Jul 21 22:27:32.694046 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:32.694040 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w" Jul 21 22:27:32.694253 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:32.694062 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a182c441bfddf3acb067871a72612ac14c5d403b0888972ecf2f8e6df79339" Jul 21 22:27:38.604690 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:38.604652 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:38.701903 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:38.701869 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:38.796007 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:38.795984 2555 scope.go:117] "RemoveContainer" containerID="998ff170c4cddc5a362effcb72efcacbbadb9c451c819971cea7837d8f11b1c9" Jul 21 22:27:44.558621 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.558579 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:44.559010 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.558801 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" podUID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" containerName="maas-api" containerID="cri-o://3ba029b0f6d512b55a725457276131694def1314388abe121030e6e3fe59b35f" gracePeriod=30 Jul 21 22:27:44.585447 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.585419 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:44.755210 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.755174 2555 generic.go:356] "Generic (PLEG): container finished" podID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" containerID="3ba029b0f6d512b55a725457276131694def1314388abe121030e6e3fe59b35f" exitCode=0 Jul 21 22:27:44.755357 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.755205 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" event={"ID":"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1","Type":"ContainerDied","Data":"3ba029b0f6d512b55a725457276131694def1314388abe121030e6e3fe59b35f"} Jul 21 22:27:44.796861 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.796836 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:44.872567 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.872540 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqgmx\" (UniqueName: \"kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx\") pod \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " Jul 21 22:27:44.872744 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.872591 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls\") pod \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " Jul 21 22:27:44.872744 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.872663 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca\") pod \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\" (UID: \"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1\") " Jul 21 22:27:44.873052 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.873026 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" (UID: "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:27:44.874706 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.874676 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" (UID: "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:27:44.874706 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.874683 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx" (OuterVolumeSpecName: "kube-api-access-cqgmx") pod "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" (UID: "c729e48e-79fd-49f0-9c2e-6f7d92c53bf1"). InnerVolumeSpecName "kube-api-access-cqgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:27:44.973790 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.973750 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:27:44.973790 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.973787 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqgmx\" (UniqueName: \"kubernetes.io/projected/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-kube-api-access-cqgmx\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:27:44.973954 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:44.973801 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:27:45.760407 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:45.760380 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" Jul 21 22:27:45.760818 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:45.760381 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb" event={"ID":"c729e48e-79fd-49f0-9c2e-6f7d92c53bf1","Type":"ContainerDied","Data":"69d5f9a7850119d60a2e46ce420aae28325f0a60c6ef1b174081ec8bbb75756d"} Jul 21 22:27:45.760818 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:45.760518 2555 scope.go:117] "RemoveContainer" containerID="3ba029b0f6d512b55a725457276131694def1314388abe121030e6e3fe59b35f" Jul 21 22:27:45.783025 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:45.782997 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:45.786751 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:45.786733 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-09bf16-6f84775479-zl5pb"] Jul 21 22:27:46.664256 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:46.664221 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" path="/var/lib/kubelet/pods/c729e48e-79fd-49f0-9c2e-6f7d92c53bf1/volumes" Jul 21 22:27:49.526079 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:49.526044 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w"] Jul 21 22:27:49.526637 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:49.526246 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:49.528422 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:49.528400 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-09bf16-kkg9w"] Jul 21 22:27:49.918321 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:49.918293 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:50.663904 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:50.663869 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5c2eeb-e754-47b7-a585-59e7c07a97f5" path="/var/lib/kubelet/pods/8b5c2eeb-e754-47b7-a585-59e7c07a97f5/volumes" Jul 21 22:27:55.795410 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:55.795379 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:55.999937 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:55.999903 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:56.099395 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.099363 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:56.261117 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.261083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:56.604976 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.604941 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:27:56.671916 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.671885 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d"] Jul 21 22:27:56.672191 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672180 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" containerName="maas-api" Jul 21 22:27:56.672242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672193 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" containerName="maas-api" Jul 21 22:27:56.672242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672206 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5c2eeb-e754-47b7-a585-59e7c07a97f5" containerName="revoke-keys" Jul 21 22:27:56.672242 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672211 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5c2eeb-e754-47b7-a585-59e7c07a97f5" containerName="revoke-keys" Jul 21 22:27:56.672332 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672267 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5c2eeb-e754-47b7-a585-59e7c07a97f5" containerName="revoke-keys" Jul 21 22:27:56.672332 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.672277 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c729e48e-79fd-49f0-9c2e-6f7d92c53bf1" containerName="maas-api" Jul 21 22:27:56.676419 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.676404 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:27:56.679187 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.679166 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:27:56.684536 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.684510 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d"] Jul 21 22:27:56.757933 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.757901 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76e244-2fm6d\" (UID: \"21555829-2d9f-4394-bfa0-a39f2f77bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:27:56.858756 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.858658 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76e244-2fm6d\" (UID: \"21555829-2d9f-4394-bfa0-a39f2f77bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:27:56.859286 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.859265 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76e244-2fm6d\" (UID: \"21555829-2d9f-4394-bfa0-a39f2f77bdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:27:56.987747 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:56.987708 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:27:57.113420 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:57.113337 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d"] Jul 21 22:27:57.116377 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:27:57.116349 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21555829_2d9f_4394_bfa0_a39f2f77bdd2.slice/crio-69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f WatchSource:0}: Error finding container 69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f: Status 404 returned error can't find the container with id 69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f Jul 21 22:27:57.815036 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:57.815000 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" event={"ID":"21555829-2d9f-4394-bfa0-a39f2f77bdd2","Type":"ContainerStarted","Data":"a8be71c1d67b7f2bb5bd463d691dd4431937b9a7df09bb3747fece49447c2a41"} Jul 21 22:27:57.815250 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:57.815063 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" event={"ID":"21555829-2d9f-4394-bfa0-a39f2f77bdd2","Type":"ContainerStarted","Data":"69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f"} Jul 21 22:27:57.833205 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:57.833159 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" podStartSLOduration=1.833142125 podStartE2EDuration="1.833142125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:27:57.83147817 +0000 UTC m=+2359.682238011" watchObservedRunningTime="2026-07-21 22:27:57.833142125 +0000 UTC m=+2359.683901955" Jul 21 22:27:58.821430 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:58.821397 2555 generic.go:356] "Generic (PLEG): container finished" podID="21555829-2d9f-4394-bfa0-a39f2f77bdd2" containerID="a8be71c1d67b7f2bb5bd463d691dd4431937b9a7df09bb3747fece49447c2a41" exitCode=0 Jul 21 22:27:58.821917 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:58.821447 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" event={"ID":"21555829-2d9f-4394-bfa0-a39f2f77bdd2","Type":"ContainerDied","Data":"a8be71c1d67b7f2bb5bd463d691dd4431937b9a7df09bb3747fece49447c2a41"} Jul 21 22:27:59.958404 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:27:59.958379 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:28:00.085151 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.085048 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca\") pod \"21555829-2d9f-4394-bfa0-a39f2f77bdd2\" (UID: \"21555829-2d9f-4394-bfa0-a39f2f77bdd2\") " Jul 21 22:28:00.085491 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.085461 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21555829-2d9f-4394-bfa0-a39f2f77bdd2" (UID: "21555829-2d9f-4394-bfa0-a39f2f77bdd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:28:00.185833 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.185800 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21555829-2d9f-4394-bfa0-a39f2f77bdd2-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:28:00.832060 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.832029 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" event={"ID":"21555829-2d9f-4394-bfa0-a39f2f77bdd2","Type":"ContainerDied","Data":"69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f"} Jul 21 22:28:00.832060 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.832050 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d" Jul 21 22:28:00.832271 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:00.832076 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69499fcd2f54c7ef57edd255ff1d2c23fb0d08c2241c2e6a9f01536fcff80b4f" Jul 21 22:28:12.085986 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.085948 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:28:12.086496 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.086210 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" podUID="12a9d170-03ae-4f34-ade4-2f46f50efac5" containerName="maas-api" containerID="cri-o://938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba" gracePeriod=30 Jul 21 22:28:12.111130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.111094 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:12.333971 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.333947 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:28:12.489432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.489348 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca\") pod \"12a9d170-03ae-4f34-ade4-2f46f50efac5\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " Jul 21 22:28:12.489432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.489407 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnkzw\" (UniqueName: \"kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw\") pod \"12a9d170-03ae-4f34-ade4-2f46f50efac5\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " Jul 21 22:28:12.489690 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.489453 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") pod \"12a9d170-03ae-4f34-ade4-2f46f50efac5\" (UID: \"12a9d170-03ae-4f34-ade4-2f46f50efac5\") " Jul 21 22:28:12.489765 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.489721 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12a9d170-03ae-4f34-ade4-2f46f50efac5" (UID: "12a9d170-03ae-4f34-ade4-2f46f50efac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:28:12.491506 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.491468 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12a9d170-03ae-4f34-ade4-2f46f50efac5" (UID: "12a9d170-03ae-4f34-ade4-2f46f50efac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:28:12.491506 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.491492 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw" (OuterVolumeSpecName: "kube-api-access-fnkzw") pod "12a9d170-03ae-4f34-ade4-2f46f50efac5" (UID: "12a9d170-03ae-4f34-ade4-2f46f50efac5"). InnerVolumeSpecName "kube-api-access-fnkzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:28:12.590082 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.590047 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a9d170-03ae-4f34-ade4-2f46f50efac5-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:28:12.590082 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.590078 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnkzw\" (UniqueName: \"kubernetes.io/projected/12a9d170-03ae-4f34-ade4-2f46f50efac5-kube-api-access-fnkzw\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:28:12.590082 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.590087 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a9d170-03ae-4f34-ade4-2f46f50efac5-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:28:12.884256 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.884225 2555 generic.go:356] "Generic (PLEG): container finished" podID="12a9d170-03ae-4f34-ade4-2f46f50efac5" containerID="938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba" exitCode=0 Jul 21 22:28:12.884421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.884265 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" event={"ID":"12a9d170-03ae-4f34-ade4-2f46f50efac5","Type":"ContainerDied","Data":"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba"} Jul 21 22:28:12.884421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.884273 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" Jul 21 22:28:12.884421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.884303 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj" event={"ID":"12a9d170-03ae-4f34-ade4-2f46f50efac5","Type":"ContainerDied","Data":"860601eef1523ae4bb06ea8f48a90cb1185af2467d1599ec9947ef441ce30957"} Jul 21 22:28:12.884421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.884324 2555 scope.go:117] "RemoveContainer" containerID="938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba" Jul 21 22:28:12.892575 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.892558 2555 scope.go:117] "RemoveContainer" containerID="938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba" Jul 21 22:28:12.892817 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:28:12.892797 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba\": container with ID starting with 938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba not found: ID does not exist" containerID="938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba" Jul 21 22:28:12.892869 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.892823 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba"} err="failed to get container status \"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba\": rpc error: code = NotFound desc = could not find container \"938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba\": container with ID starting with 938e026ad1fdb98e1ba0ab08d59f1c3986790efef76c512d1d746fce194e3bba not found: ID does not exist" Jul 21 22:28:12.901054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.901029 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:28:12.909739 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:12.909715 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-76e244-84fdc9c64b-h8xdj"] Jul 21 22:28:14.663970 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:14.663937 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a9d170-03ae-4f34-ade4-2f46f50efac5" path="/var/lib/kubelet/pods/12a9d170-03ae-4f34-ade4-2f46f50efac5/volumes" Jul 21 22:28:15.059098 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:15.059016 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:16.703839 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:16.703795 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d"] Jul 21 22:28:16.706624 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:16.706584 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76e244-2fm6d"] Jul 21 22:28:16.706872 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:16.706845 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:17.108591 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:17.108559 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:18.664589 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:18.664502 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21555829-2d9f-4394-bfa0-a39f2f77bdd2" path="/var/lib/kubelet/pods/21555829-2d9f-4394-bfa0-a39f2f77bdd2/volumes" Jul 21 22:28:22.286374 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:22.286336 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:22.387114 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:22.387086 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:23.298479 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:23.298442 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:23.366437 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:23.366402 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:23.595759 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:23.595733 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:23.825063 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:23.825033 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:24.218189 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:24.218145 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:24.553982 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:24.553895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:25.044235 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:25.044189 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:25.831708 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:25.831674 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:27.302439 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.302409 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:27.400856 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.400815 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:28:27.401284 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401269 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a9d170-03ae-4f34-ade4-2f46f50efac5" containerName="maas-api" Jul 21 22:28:27.401325 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401288 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a9d170-03ae-4f34-ade4-2f46f50efac5" containerName="maas-api" Jul 21 22:28:27.401325 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401312 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21555829-2d9f-4394-bfa0-a39f2f77bdd2" containerName="revoke-keys" Jul 21 22:28:27.401325 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401320 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="21555829-2d9f-4394-bfa0-a39f2f77bdd2" containerName="revoke-keys" Jul 21 22:28:27.401421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401398 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a9d170-03ae-4f34-ade4-2f46f50efac5" containerName="maas-api" Jul 21 22:28:27.401421 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.401413 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="21555829-2d9f-4394-bfa0-a39f2f77bdd2" containerName="revoke-keys" Jul 21 22:28:27.404807 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.404785 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.405355 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.405334 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:27.407382 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.407361 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-131f31-serving-cert\"" Jul 21 22:28:27.407500 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.407463 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fwz59\"" Jul 21 22:28:27.414943 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.414915 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:28:27.481062 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.481023 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:27.511169 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.511128 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hf9k\" (UniqueName: \"kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.511346 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.511175 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.511346 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.511317 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.612701 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.612656 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.612927 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.612741 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hf9k\" (UniqueName: \"kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.612927 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.612779 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.613435 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.613410 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.614988 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.614970 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.622051 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.622025 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hf9k\" (UniqueName: \"kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k\") pod \"maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.631409 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.631368 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:27.717898 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.717860 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:27.850986 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.850959 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:28:27.853024 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:28:27.852995 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92f3537_cb81_437a_a6be_6c071e04d1ff.slice/crio-3717a37a8b367c057a8d2bb5fa7e85fcbcba824c44fc4df56be1f95f68a2f296 WatchSource:0}: Error finding container 3717a37a8b367c057a8d2bb5fa7e85fcbcba824c44fc4df56be1f95f68a2f296: Status 404 returned error can't find the container with id 3717a37a8b367c057a8d2bb5fa7e85fcbcba824c44fc4df56be1f95f68a2f296 Jul 21 22:28:27.955779 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:27.955668 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" event={"ID":"e92f3537-cb81-437a-a6be-6c071e04d1ff","Type":"ContainerStarted","Data":"3717a37a8b367c057a8d2bb5fa7e85fcbcba824c44fc4df56be1f95f68a2f296"} Jul 21 22:28:28.031572 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:28.031534 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:28.131809 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:28.131771 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:28.961817 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:28.961779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" event={"ID":"e92f3537-cb81-437a-a6be-6c071e04d1ff","Type":"ContainerStarted","Data":"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b"} Jul 21 22:28:28.962260 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:28.961858 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:28.985420 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:28.985373 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" podStartSLOduration=1.442279473 podStartE2EDuration="1.985357092s" totalImagesPullingTime="543.077619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:28:28.984476311 +0000 UTC m=+2390.835236141" watchObservedRunningTime="2026-07-21 22:28:28.985357092 +0000 UTC m=+2390.836116926" Jul 21 22:28:31.803664 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:31.803586 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:32.137131 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:32.137102 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:32.484172 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:32.484083 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:32.547358 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:32.547328 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:34.971655 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:34.971597 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:28:35.341358 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.341321 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:35.441699 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.441667 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:35.452649 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.452586 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:28:35.456703 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.456677 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.459436 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.459410 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b77603-serving-cert\"" Jul 21 22:28:35.467796 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.467731 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:28:35.542492 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.542455 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:35.581032 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.581001 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.581212 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.581069 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9vw\" (UniqueName: \"kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.581212 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.581141 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.644350 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.644266 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:35.681601 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.681562 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9vw\" (UniqueName: \"kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.681601 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.681618 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.681876 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.681662 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.682395 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.682371 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.684042 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.684024 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.690429 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.690407 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9vw\" (UniqueName: \"kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw\") pod \"maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.770388 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.770350 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:35.903110 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.903029 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:28:35.906558 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:28:35.906532 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5589e994_e85e_498f_8747_e13940561f69.slice/crio-862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e WatchSource:0}: Error finding container 862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e: Status 404 returned error can't find the container with id 862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e Jul 21 22:28:35.994042 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:35.994003 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" event={"ID":"5589e994-e85e-498f-8747-e13940561f69","Type":"ContainerStarted","Data":"862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e"} Jul 21 22:28:36.043623 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:36.043580 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:36.144056 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:36.144011 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:37.000713 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:37.000683 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" event={"ID":"5589e994-e85e-498f-8747-e13940561f69","Type":"ContainerStarted","Data":"98283c258c2dcde559e7363115fd0052a994a8dd094e19053a2a6878c9f19489"} Jul 21 22:28:37.001119 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:37.000745 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:37.018172 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:37.018081 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" podStartSLOduration=1.432260842 podStartE2EDuration="2.018058789s" totalImagesPullingTime="585.797947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:28:37.017124405 +0000 UTC m=+2398.867884234" watchObservedRunningTime="2026-07-21 22:28:37.018058789 +0000 UTC m=+2398.868818618" Jul 21 22:28:38.819351 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:38.819318 2555 scope.go:117] "RemoveContainer" containerID="d6b8d85ccb4aae490550dbf423f7b8245014674ec6357dea2f4f6da2278e528f" Jul 21 22:28:39.837299 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:39.837255 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:43.011275 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:43.011242 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:28:48.772533 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:48.772497 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:48.872367 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:48.872316 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:53.526982 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:53.526946 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:56.831091 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:56.831057 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:28:56.932041 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:28:56.932003 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:29:20.006970 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:29:20.006934 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:29:38.203987 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:29:38.203947 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:30:00.138681 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.138648 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5"] Jul 21 22:30:00.141897 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.141880 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" Jul 21 22:30:00.144982 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.144955 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:30:00.145391 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.145372 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6"] Jul 21 22:30:00.149023 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.148899 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2"] Jul 21 22:30:00.149136 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.149117 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" Jul 21 22:30:00.151731 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.151715 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" Jul 21 22:30:00.152109 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.152093 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" Jul 21 22:30:00.155749 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.155730 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5"] Jul 21 22:30:00.158112 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.158061 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6"] Jul 21 22:30:00.160800 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.160774 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2"] Jul 21 22:30:00.164193 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.164172 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" Jul 21 22:30:00.207330 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.207302 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" Jul 21 22:30:00.315033 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.314954 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5"] Jul 21 22:30:00.342768 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.342747 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6"] Jul 21 22:30:00.346038 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:30:00.346013 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade0ec87_f0f6_46ce_a051_5e02913c2216.slice/crio-31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363 WatchSource:0}: Error finding container 31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363: Status 404 returned error can't find the container with id 31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363 Jul 21 22:30:00.373522 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.373500 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2"] Jul 21 22:30:00.381758 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:30:00.381732 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8cc0d5_515b_4417_a84c_93a5afa7821e.slice/crio-bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0 WatchSource:0}: Error finding container bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0: Status 404 returned error can't find the container with id bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0 Jul 21 22:30:00.383133 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.383109 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" event={"ID":"ade0ec87-f0f6-46ce-a051-5e02913c2216","Type":"ContainerStarted","Data":"31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363"} Jul 21 22:30:00.384272 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:00.384247 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" event={"ID":"bbad6202-6cf2-4b42-a49d-79ccdce59c4b","Type":"ContainerStarted","Data":"ec6c9a30a766bd92644f2b07bf34439c3c4b76cfc9fbdb71da6a896da9c9749a"} Jul 21 22:30:01.390284 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.390234 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" event={"ID":"ade0ec87-f0f6-46ce-a051-5e02913c2216","Type":"ContainerStarted","Data":"37a26d9710ee974c96776bdaca7215e3de59ba385691e4a181f896ce128eacf6"} Jul 21 22:30:01.391725 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.391701 2555 generic.go:356] "Generic (PLEG): container finished" podID="bbad6202-6cf2-4b42-a49d-79ccdce59c4b" containerID="736814f01e42636d765d9c01d31f225f7a31418e274a1df21061bdeae8993c93" exitCode=0 Jul 21 22:30:01.391847 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.391779 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" event={"ID":"bbad6202-6cf2-4b42-a49d-79ccdce59c4b","Type":"ContainerDied","Data":"736814f01e42636d765d9c01d31f225f7a31418e274a1df21061bdeae8993c93"} Jul 21 22:30:01.393201 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.393179 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" event={"ID":"bf8cc0d5-515b-4417-a84c-93a5afa7821e","Type":"ContainerStarted","Data":"da6e97a75da928ba3d7a1f92a92ba07736b67f327f2b1c875d32941861109c9e"} Jul 21 22:30:01.393307 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.393215 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" event={"ID":"bf8cc0d5-515b-4417-a84c-93a5afa7821e","Type":"ContainerStarted","Data":"bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0"} Jul 21 22:30:01.407809 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.407760 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" podStartSLOduration=1.407746345 podStartE2EDuration="1.407746345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:30:01.407383377 +0000 UTC m=+2483.258143220" watchObservedRunningTime="2026-07-21 22:30:01.407746345 +0000 UTC m=+2483.258506176" Jul 21 22:30:01.440927 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:01.440885 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" podStartSLOduration=1.440871508 podStartE2EDuration="1.440871508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:30:01.439464194 +0000 UTC m=+2483.290224022" watchObservedRunningTime="2026-07-21 22:30:01.440871508 +0000 UTC m=+2483.291631338" Jul 21 22:30:02.399234 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:02.399202 2555 generic.go:356] "Generic (PLEG): container finished" podID="bf8cc0d5-515b-4417-a84c-93a5afa7821e" containerID="da6e97a75da928ba3d7a1f92a92ba07736b67f327f2b1c875d32941861109c9e" exitCode=0 Jul 21 22:30:02.399730 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:02.399290 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" event={"ID":"bf8cc0d5-515b-4417-a84c-93a5afa7821e","Type":"ContainerDied","Data":"da6e97a75da928ba3d7a1f92a92ba07736b67f327f2b1c875d32941861109c9e"} Jul 21 22:30:02.400713 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:02.400694 2555 generic.go:356] "Generic (PLEG): container finished" podID="ade0ec87-f0f6-46ce-a051-5e02913c2216" containerID="37a26d9710ee974c96776bdaca7215e3de59ba385691e4a181f896ce128eacf6" exitCode=0 Jul 21 22:30:02.400786 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:02.400739 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" event={"ID":"ade0ec87-f0f6-46ce-a051-5e02913c2216","Type":"ContainerDied","Data":"37a26d9710ee974c96776bdaca7215e3de59ba385691e4a181f896ce128eacf6"} Jul 21 22:30:02.531545 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:02.531519 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" Jul 21 22:30:03.407161 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:03.407132 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" Jul 21 22:30:03.407161 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:03.407153 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744550-7q9w5" event={"ID":"bbad6202-6cf2-4b42-a49d-79ccdce59c4b","Type":"ContainerDied","Data":"ec6c9a30a766bd92644f2b07bf34439c3c4b76cfc9fbdb71da6a896da9c9749a"} Jul 21 22:30:03.407700 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:03.407204 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6c9a30a766bd92644f2b07bf34439c3c4b76cfc9fbdb71da6a896da9c9749a" Jul 21 22:30:03.533158 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:03.533136 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" Jul 21 22:30:03.567576 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:03.567548 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" Jul 21 22:30:04.415805 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.415773 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" Jul 21 22:30:04.416249 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.415810 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2" event={"ID":"bf8cc0d5-515b-4417-a84c-93a5afa7821e","Type":"ContainerDied","Data":"bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0"} Jul 21 22:30:04.416249 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.415866 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde30d223566d17109c6dc139e2a4230e7a44adbded7f2f91b5ed0aef37fc2d0" Jul 21 22:30:04.417157 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.417132 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" Jul 21 22:30:04.417264 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.417133 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6" event={"ID":"ade0ec87-f0f6-46ce-a051-5e02913c2216","Type":"ContainerDied","Data":"31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363"} Jul 21 22:30:04.417264 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:30:04.417250 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31a4bec82c56d870144bd1cdfc58ef7603778037cb8a5396af8ae026b2acf363" Jul 21 22:31:05.531886 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:05.531853 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:05.588980 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:05.588950 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:15.683641 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:15.683594 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:19.213849 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:19.213816 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:20.250751 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:20.250719 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:34.142848 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:34.142815 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:31:58.672624 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:31:58.672575 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:32:38.867922 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:32:38.867894 2555 scope.go:117] "RemoveContainer" containerID="6dc045a28d2725937fc57db7ce672933e6881e3f12cfdcd623e9adca7a25ba71" Jul 21 22:32:45.391267 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:32:45.391231 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:32:47.626065 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:32:47.626031 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:32:48.406057 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:32:48.406025 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:32:51.896438 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:32:51.896405 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:00.919973 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:00.919935 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:20.007265 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:20.007227 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:23.227315 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:23.227267 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:23.330432 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:23.330399 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:38.886043 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:38.886014 2555 scope.go:117] "RemoveContainer" containerID="c0e911793c15992b61c7d22653ee6b0d8f204d7fc189184bd2af1fd73977bd54" Jul 21 22:33:38.893902 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:38.893877 2555 scope.go:117] "RemoveContainer" containerID="89f3af106926027b3ba48ff9f136a732387cef73d07ee98342747bee033db0ea" Jul 21 22:33:49.585302 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:49.585262 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:49.685974 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:49.685943 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:57.667069 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:57.667039 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:33:57.772148 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:33:57.772117 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:02.724440 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:02.724401 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:03.516743 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:03.516714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:22.722524 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722490 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm"] Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722851 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf8cc0d5-515b-4417-a84c-93a5afa7821e" containerName="cleanup" Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722864 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf8cc0d5-515b-4417-a84c-93a5afa7821e" containerName="cleanup" Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722872 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade0ec87-f0f6-46ce-a051-5e02913c2216" containerName="cleanup" Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722879 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade0ec87-f0f6-46ce-a051-5e02913c2216" containerName="cleanup" Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722886 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbad6202-6cf2-4b42-a49d-79ccdce59c4b" containerName="cleanup" Jul 21 22:34:22.722915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722891 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbad6202-6cf2-4b42-a49d-79ccdce59c4b" containerName="cleanup" Jul 21 22:34:22.723109 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722946 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbad6202-6cf2-4b42-a49d-79ccdce59c4b" containerName="cleanup" Jul 21 22:34:22.723109 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722956 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf8cc0d5-515b-4417-a84c-93a5afa7821e" containerName="cleanup" Jul 21 22:34:22.723109 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.722962 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade0ec87-f0f6-46ce-a051-5e02913c2216" containerName="cleanup" Jul 21 22:34:22.726083 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.726066 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:22.728751 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.728731 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:34:22.733249 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.733044 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm"] Jul 21 22:34:22.835071 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.835036 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm\" (UID: \"91e64cd3-0adc-431f-a234-d2e7a82f6d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:22.936079 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.936047 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm\" (UID: \"91e64cd3-0adc-431f-a234-d2e7a82f6d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:22.936779 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:22.936754 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm\" (UID: \"91e64cd3-0adc-431f-a234-d2e7a82f6d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:23.037207 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.037131 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:23.168270 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.168249 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm"] Jul 21 22:34:23.172681 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:34:23.172646 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e64cd3_0adc_431f_a234_d2e7a82f6d29.slice/crio-429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa WatchSource:0}: Error finding container 429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa: Status 404 returned error can't find the container with id 429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa Jul 21 22:34:23.389553 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.389522 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:23.491933 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.491893 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:23.620117 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.620027 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" event={"ID":"91e64cd3-0adc-431f-a234-d2e7a82f6d29","Type":"ContainerStarted","Data":"50e78f335070caf18d765e69401f515336c6aa06db9e723368318c62ccb6284a"} Jul 21 22:34:23.620117 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:23.620108 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" event={"ID":"91e64cd3-0adc-431f-a234-d2e7a82f6d29","Type":"ContainerStarted","Data":"429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa"} Jul 21 22:34:24.626398 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:24.626366 2555 generic.go:356] "Generic (PLEG): container finished" podID="91e64cd3-0adc-431f-a234-d2e7a82f6d29" containerID="50e78f335070caf18d765e69401f515336c6aa06db9e723368318c62ccb6284a" exitCode=0 Jul 21 22:34:24.626834 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:24.626459 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" event={"ID":"91e64cd3-0adc-431f-a234-d2e7a82f6d29","Type":"ContainerDied","Data":"50e78f335070caf18d765e69401f515336c6aa06db9e723368318c62ccb6284a"} Jul 21 22:34:25.761289 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:25.761263 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:25.860459 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:25.860423 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca\") pod \"91e64cd3-0adc-431f-a234-d2e7a82f6d29\" (UID: \"91e64cd3-0adc-431f-a234-d2e7a82f6d29\") " Jul 21 22:34:25.860805 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:25.860782 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91e64cd3-0adc-431f-a234-d2e7a82f6d29" (UID: "91e64cd3-0adc-431f-a234-d2e7a82f6d29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:34:25.961633 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:25.961527 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91e64cd3-0adc-431f-a234-d2e7a82f6d29-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:26.636437 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:26.636408 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" Jul 21 22:34:26.636437 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:26.636414 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm" event={"ID":"91e64cd3-0adc-431f-a234-d2e7a82f6d29","Type":"ContainerDied","Data":"429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa"} Jul 21 22:34:26.636670 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:26.636454 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="429c06d8280a144f5d67733ff121d7ebc84132f8e7f5c2885d8819afd56820aa" Jul 21 22:34:33.410483 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.410446 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:34:33.411268 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.411237 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" podUID="e92f3537-cb81-437a-a6be-6c071e04d1ff" containerName="maas-api" containerID="cri-o://853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b" gracePeriod=30 Jul 21 22:34:33.441966 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.441937 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:33.450089 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.450066 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2"] Jul 21 22:34:33.452987 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.452966 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-131f31-29744550-6mxs2"] Jul 21 22:34:33.659752 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.659727 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:34:33.668645 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.668566 2555 generic.go:356] "Generic (PLEG): container finished" podID="e92f3537-cb81-437a-a6be-6c071e04d1ff" containerID="853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b" exitCode=0 Jul 21 22:34:33.668645 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.668601 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" Jul 21 22:34:33.668769 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.668643 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" event={"ID":"e92f3537-cb81-437a-a6be-6c071e04d1ff","Type":"ContainerDied","Data":"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b"} Jul 21 22:34:33.668769 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.668698 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9" event={"ID":"e92f3537-cb81-437a-a6be-6c071e04d1ff","Type":"ContainerDied","Data":"3717a37a8b367c057a8d2bb5fa7e85fcbcba824c44fc4df56be1f95f68a2f296"} Jul 21 22:34:33.668769 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.668722 2555 scope.go:117] "RemoveContainer" containerID="853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b" Jul 21 22:34:33.677594 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.677575 2555 scope.go:117] "RemoveContainer" containerID="853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b" Jul 21 22:34:33.677882 ip-10-0-130-224 kubenswrapper[2555]: E0721 22:34:33.677863 2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b\": container with ID starting with 853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b not found: ID does not exist" containerID="853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b" Jul 21 22:34:33.677945 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.677889 2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b"} err="failed to get container status \"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b\": rpc error: code = NotFound desc = could not find container \"853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b\": container with ID starting with 853bdfa7c2659f9ba2b525a34205158bfb10a6dc93442eace2e4fcb26474d47b not found: ID does not exist" Jul 21 22:34:33.725730 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.725702 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hf9k\" (UniqueName: \"kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k\") pod \"e92f3537-cb81-437a-a6be-6c071e04d1ff\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " Jul 21 22:34:33.725866 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.725756 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls\") pod \"e92f3537-cb81-437a-a6be-6c071e04d1ff\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " Jul 21 22:34:33.725866 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.725832 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca\") pod \"e92f3537-cb81-437a-a6be-6c071e04d1ff\" (UID: \"e92f3537-cb81-437a-a6be-6c071e04d1ff\") " Jul 21 22:34:33.726189 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.726165 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e92f3537-cb81-437a-a6be-6c071e04d1ff" (UID: "e92f3537-cb81-437a-a6be-6c071e04d1ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:34:33.727867 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.727842 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e92f3537-cb81-437a-a6be-6c071e04d1ff" (UID: "e92f3537-cb81-437a-a6be-6c071e04d1ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:34:33.727953 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.727911 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k" (OuterVolumeSpecName: "kube-api-access-7hf9k") pod "e92f3537-cb81-437a-a6be-6c071e04d1ff" (UID: "e92f3537-cb81-437a-a6be-6c071e04d1ff"). InnerVolumeSpecName "kube-api-access-7hf9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:33.826824 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.826798 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92f3537-cb81-437a-a6be-6c071e04d1ff-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:33.826824 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.826820 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hf9k\" (UniqueName: \"kubernetes.io/projected/e92f3537-cb81-437a-a6be-6c071e04d1ff-kube-api-access-7hf9k\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:33.826964 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.826830 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92f3537-cb81-437a-a6be-6c071e04d1ff-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:33.992293 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.992258 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:34:33.996362 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:33.996341 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-131f31-5c4b697fd7-9skl9"] Jul 21 22:34:34.663601 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:34.663572 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf8cc0d5-515b-4417-a84c-93a5afa7821e" path="/var/lib/kubelet/pods/bf8cc0d5-515b-4417-a84c-93a5afa7821e/volumes" Jul 21 22:34:34.663955 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:34.663920 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92f3537-cb81-437a-a6be-6c071e04d1ff" path="/var/lib/kubelet/pods/e92f3537-cb81-437a-a6be-6c071e04d1ff/volumes" Jul 21 22:34:37.797002 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:37.796960 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:37.828021 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:37.827958 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm"] Jul 21 22:34:37.830024 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:37.830001 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-131f31-7x5vm"] Jul 21 22:34:38.216957 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:38.216929 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:38.664800 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:38.664769 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e64cd3-0adc-431f-a234-d2e7a82f6d29" path="/var/lib/kubelet/pods/91e64cd3-0adc-431f-a234-d2e7a82f6d29/volumes" Jul 21 22:34:38.914721 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:38.914693 2555 scope.go:117] "RemoveContainer" containerID="a8be71c1d67b7f2bb5bd463d691dd4431937b9a7df09bb3747fece49447c2a41" Jul 21 22:34:44.425841 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:44.425804 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:44.526930 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:44.526895 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:44.691939 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:44.691867 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:44.957231 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:44.957149 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:45.092288 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092257 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2"] Jul 21 22:34:45.092588 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092576 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92f3537-cb81-437a-a6be-6c071e04d1ff" containerName="maas-api" Jul 21 22:34:45.092655 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092590 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92f3537-cb81-437a-a6be-6c071e04d1ff" containerName="maas-api" Jul 21 22:34:45.092655 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092598 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e64cd3-0adc-431f-a234-d2e7a82f6d29" containerName="revoke-keys" Jul 21 22:34:45.092655 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092617 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e64cd3-0adc-431f-a234-d2e7a82f6d29" containerName="revoke-keys" Jul 21 22:34:45.092750 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092667 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e64cd3-0adc-431f-a234-d2e7a82f6d29" containerName="revoke-keys" Jul 21 22:34:45.092750 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.092677 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92f3537-cb81-437a-a6be-6c071e04d1ff" containerName="maas-api" Jul 21 22:34:45.096883 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.096865 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:45.100275 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.100253 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m8dwg\"" Jul 21 22:34:45.108800 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.108778 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2"] Jul 21 22:34:45.218825 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.218760 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b77603-98bh2\" (UID: \"9106b7a9-af53-4f07-84d7-f958f3ba6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:45.320112 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.320083 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b77603-98bh2\" (UID: \"9106b7a9-af53-4f07-84d7-f958f3ba6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:45.320632 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.320593 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b77603-98bh2\" (UID: \"9106b7a9-af53-4f07-84d7-f958f3ba6028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:45.406997 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.406974 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:45.535369 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.535343 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2"] Jul 21 22:34:45.537843 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:34:45.537810 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9106b7a9_af53_4f07_84d7_f958f3ba6028.slice/crio-948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b WatchSource:0}: Error finding container 948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b: Status 404 returned error can't find the container with id 948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b Jul 21 22:34:45.724925 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.724860 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:45.729703 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.729674 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" event={"ID":"9106b7a9-af53-4f07-84d7-f958f3ba6028","Type":"ContainerStarted","Data":"948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b"} Jul 21 22:34:45.825913 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:45.825885 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:46.735813 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:46.735777 2555 generic.go:356] "Generic (PLEG): container finished" podID="9106b7a9-af53-4f07-84d7-f958f3ba6028" containerID="711397cf4f24a8bc84a89f2be25f007e58067dc5c7f0e65ed5978d28daf6029b" exitCode=0 Jul 21 22:34:46.736221 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:46.735865 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" event={"ID":"9106b7a9-af53-4f07-84d7-f958f3ba6028","Type":"ContainerDied","Data":"711397cf4f24a8bc84a89f2be25f007e58067dc5c7f0e65ed5978d28daf6029b"} Jul 21 22:34:47.872408 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:47.872386 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:47.939737 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:47.939712 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca\") pod \"9106b7a9-af53-4f07-84d7-f958f3ba6028\" (UID: \"9106b7a9-af53-4f07-84d7-f958f3ba6028\") " Jul 21 22:34:47.940034 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:47.940014 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9106b7a9-af53-4f07-84d7-f958f3ba6028" (UID: "9106b7a9-af53-4f07-84d7-f958f3ba6028"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:34:48.041141 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:48.041087 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9106b7a9-af53-4f07-84d7-f958f3ba6028-maas-api-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:48.746000 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:48.745973 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" Jul 21 22:34:48.746000 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:48.745988 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2" event={"ID":"9106b7a9-af53-4f07-84d7-f958f3ba6028","Type":"ContainerDied","Data":"948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b"} Jul 21 22:34:48.746170 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:48.746030 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948e91ef19a26578af9cddaed0dff83cfb018a83734c8ec34255175be45cb14b" Jul 21 22:34:55.538409 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.538365 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:34:55.538937 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.538640 2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" podUID="5589e994-e85e-498f-8747-e13940561f69" containerName="maas-api" containerID="cri-o://98283c258c2dcde559e7363115fd0052a994a8dd094e19053a2a6878c9f19489" gracePeriod=30 Jul 21 22:34:55.566148 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.566121 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:34:55.575661 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.575596 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6"] Jul 21 22:34:55.577694 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.577673 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b77603-29744550-p49n6"] Jul 21 22:34:55.780624 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.780577 2555 generic.go:356] "Generic (PLEG): container finished" podID="5589e994-e85e-498f-8747-e13940561f69" containerID="98283c258c2dcde559e7363115fd0052a994a8dd094e19053a2a6878c9f19489" exitCode=0 Jul 21 22:34:55.780718 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.780635 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" event={"ID":"5589e994-e85e-498f-8747-e13940561f69","Type":"ContainerDied","Data":"98283c258c2dcde559e7363115fd0052a994a8dd094e19053a2a6878c9f19489"} Jul 21 22:34:55.780718 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.780685 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" event={"ID":"5589e994-e85e-498f-8747-e13940561f69","Type":"ContainerDied","Data":"862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e"} Jul 21 22:34:55.780718 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.780704 2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862dcf135ac6aa819ac943f5fc21b7416c6197750d64bec2494ee0343c1b666e" Jul 21 22:34:55.782623 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.782591 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:34:55.903970 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.903944 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca\") pod \"5589e994-e85e-498f-8747-e13940561f69\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " Jul 21 22:34:55.904130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.903999 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb9vw\" (UniqueName: \"kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw\") pod \"5589e994-e85e-498f-8747-e13940561f69\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " Jul 21 22:34:55.904130 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.904022 2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls\") pod \"5589e994-e85e-498f-8747-e13940561f69\" (UID: \"5589e994-e85e-498f-8747-e13940561f69\") " Jul 21 22:34:55.904256 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.904233 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5589e994-e85e-498f-8747-e13940561f69" (UID: "5589e994-e85e-498f-8747-e13940561f69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 22:34:55.905938 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.905913 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5589e994-e85e-498f-8747-e13940561f69" (UID: "5589e994-e85e-498f-8747-e13940561f69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:34:55.906045 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:55.905991 2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw" (OuterVolumeSpecName: "kube-api-access-rb9vw") pod "5589e994-e85e-498f-8747-e13940561f69" (UID: "5589e994-e85e-498f-8747-e13940561f69"). InnerVolumeSpecName "kube-api-access-rb9vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:56.005085 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.005062 2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5589e994-e85e-498f-8747-e13940561f69-openshift-service-ca\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:56.005085 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.005083 2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9vw\" (UniqueName: \"kubernetes.io/projected/5589e994-e85e-498f-8747-e13940561f69-kube-api-access-rb9vw\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:56.005244 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.005093 2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5589e994-e85e-498f-8747-e13940561f69-maas-api-tls\") on node \"ip-10-0-130-224.ec2.internal\" DevicePath \"\"" Jul 21 22:34:56.664553 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.664523 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade0ec87-f0f6-46ce-a051-5e02913c2216" path="/var/lib/kubelet/pods/ade0ec87-f0f6-46ce-a051-5e02913c2216/volumes" Jul 21 22:34:56.785290 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.785260 2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr" Jul 21 22:34:56.804598 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.804570 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:34:56.806153 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:56.806133 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b77603-647bb9d996-2qnpr"] Jul 21 22:34:58.664018 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:34:58.663983 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5589e994-e85e-498f-8747-e13940561f69" path="/var/lib/kubelet/pods/5589e994-e85e-498f-8747-e13940561f69/volumes" Jul 21 22:35:00.165915 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:00.165812 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:00.166340 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:00.165946 2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2"] Jul 21 22:35:00.168584 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:00.168557 2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b77603-98bh2"] Jul 21 22:35:00.558694 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:00.558578 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:00.664695 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:00.664670 2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9106b7a9-af53-4f07-84d7-f958f3ba6028" path="/var/lib/kubelet/pods/9106b7a9-af53-4f07-84d7-f958f3ba6028/volumes" Jul 21 22:35:06.753740 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:06.753714 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:06.833237 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:06.833203 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:07.085729 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:07.085696 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:07.315984 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:07.315950 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:13.347146 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:13.347113 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-f5tbw_1af52da3-84bd-4c78-8b5f-ce023a1a2554/manager/0.log" Jul 21 22:35:13.950554 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:13.950528 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-dgq64_f3b6e149-0a77-4fd9-8a81-ca2e4763c237/copy-manifests/0.log" Jul 21 22:35:13.958379 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:13.958353 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-dgq64_f3b6e149-0a77-4fd9-8a81-ca2e4763c237/manager/0.log" Jul 21 22:35:14.459820 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:14.459792 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-55794f648-fs75v_18b451fa-e9cf-44dc-a8a6-62fdc99f5984/maas-api/0.log" Jul 21 22:35:14.579531 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:14.579501 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744520-4kzp8_e30fdc68-8adf-4ad9-b378-b218fdd24b7f/cleanup/0.log" Jul 21 22:35:14.691769 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:14.691744 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744535-nwbsx_b5182e35-f604-421d-ab6a-4caccde0e368/cleanup/0.log" Jul 21 22:35:14.802054 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:14.801983 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744550-7q9w5_bbad6202-6cf2-4b42-a49d-79ccdce59c4b/cleanup/0.log" Jul 21 22:35:16.361291 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:16.361261 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hq8fk_5092bd18-788a-40ae-a44a-96efb792426a/manager/0.log" Jul 21 22:35:16.474526 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:16.474502 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zpx2w_2436d5fe-0ef7-41df-b8a6-ccd215217b93/manager/0.log" Jul 21 22:35:17.088540 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:17.088505 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlkmw_f5398776-cd1b-4e44-8791-98ea74723b22/manager/0.log" Jul 21 22:35:17.574042 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:17.574015 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4n2wk_366c5814-8054-4caf-bab0-c8633d3959e2/discovery/0.log" Jul 21 22:35:17.694247 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:17.694222 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-68vxd_5d01a020-ddca-4912-9446-3dd9e3691913/kube-auth-proxy/0.log" Jul 21 22:35:18.793291 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:18.793253 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-f8cfg_92640ae1-df20-42bc-9180-89a937874eb4/main/0.log" Jul 21 22:35:18.912010 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:18.911983 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-cgmdr_66c52393-890f-4727-952d-501b3a1ad3f3/main/0.log" Jul 21 22:35:24.843976 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:24.843930 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:24.934111 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:24.934084 2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" limit=1024 Jul 21 22:35:27.867929 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:27.867900 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tfqbs_0298deea-f91e-4d59-9170-a1914c0e4491/global-pull-secret-syncer/0.log" Jul 21 22:35:27.930680 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:27.930652 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dsqzk_6171ab4d-bb3d-4bf3-95f8-2722bc2e83c7/konnectivity-agent/0.log" Jul 21 22:35:28.027183 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:28.027151 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-224.ec2.internal_b3ed946aa6fec2dea4a6cd621aaa10c4/haproxy/0.log" Jul 21 22:35:32.483443 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:32.483411 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hq8fk_5092bd18-788a-40ae-a44a-96efb792426a/manager/0.log" Jul 21 22:35:32.512349 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:32.512328 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zpx2w_2436d5fe-0ef7-41df-b8a6-ccd215217b93/manager/0.log" Jul 21 22:35:32.850249 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:32.850222 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlkmw_f5398776-cd1b-4e44-8791-98ea74723b22/manager/0.log" Jul 21 22:35:35.140519 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:35.140491 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhssk_8043d00d-884b-4252-9c24-fb2ec17f1a70/node-exporter/0.log" Jul 21 22:35:35.164047 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:35.163986 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhssk_8043d00d-884b-4252-9c24-fb2ec17f1a70/kube-rbac-proxy/0.log" Jul 21 22:35:35.188483 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:35.188461 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhssk_8043d00d-884b-4252-9c24-fb2ec17f1a70/init-textfile/0.log" Jul 21 22:35:37.131241 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:37.131208 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-4bcf4_a5fa6eff-8f54-4c83-a2ac-a5006592dfa2/networking-console-plugin/0.log" Jul 21 22:35:38.045085 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045054 2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r"] Jul 21 22:35:38.045401 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045388 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9106b7a9-af53-4f07-84d7-f958f3ba6028" containerName="revoke-keys" Jul 21 22:35:38.045401 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045402 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9106b7a9-af53-4f07-84d7-f958f3ba6028" containerName="revoke-keys" Jul 21 22:35:38.045478 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045413 2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5589e994-e85e-498f-8747-e13940561f69" containerName="maas-api" Jul 21 22:35:38.045478 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045418 2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5589e994-e85e-498f-8747-e13940561f69" containerName="maas-api" Jul 21 22:35:38.045478 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045462 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9106b7a9-af53-4f07-84d7-f958f3ba6028" containerName="revoke-keys" Jul 21 22:35:38.045478 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.045472 2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5589e994-e85e-498f-8747-e13940561f69" containerName="maas-api" Jul 21 22:35:38.048595 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.048573 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.051237 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.051217 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"kube-root-ca.crt\"" Jul 21 22:35:38.052020 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.052002 2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wffls\"/\"default-dockercfg-fk87s\"" Jul 21 22:35:38.052081 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.052061 2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"openshift-service-ca.crt\"" Jul 21 22:35:38.058788 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.058770 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r"] Jul 21 22:35:38.114934 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.114914 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-proc\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.115037 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.114955 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqdt\" (UniqueName: \"kubernetes.io/projected/31c90d98-c368-4d38-b998-2807c1d5b8a2-kube-api-access-nxqdt\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.115037 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.114981 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-podres\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.115037 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.115008 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-sys\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.115037 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.115025 2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-lib-modules\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.215954 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.215932 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-podres\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.215967 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-sys\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.215983 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-lib-modules\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216005 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-proc\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216040 2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqdt\" (UniqueName: \"kubernetes.io/projected/31c90d98-c368-4d38-b998-2807c1d5b8a2-kube-api-access-nxqdt\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216075 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-sys\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216098 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-proc\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216119 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-lib-modules\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.216282 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.216116 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/31c90d98-c368-4d38-b998-2807c1d5b8a2-podres\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.225297 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.225237 2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqdt\" (UniqueName: \"kubernetes.io/projected/31c90d98-c368-4d38-b998-2807c1d5b8a2-kube-api-access-nxqdt\") pod \"perf-node-gather-daemonset-z9f6r\" (UID: \"31c90d98-c368-4d38-b998-2807c1d5b8a2\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.359168 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.359145 2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.487752 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.487731 2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r"] Jul 21 22:35:38.490370 ip-10-0-130-224 kubenswrapper[2555]: W0721 22:35:38.490351 2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31c90d98_c368_4d38_b998_2807c1d5b8a2.slice/crio-d400cc687f0e01b92e31d60f4fc9c06ec873227f815ba94f9aa332f622ce0627 WatchSource:0}: Error finding container d400cc687f0e01b92e31d60f4fc9c06ec873227f815ba94f9aa332f622ce0627: Status 404 returned error can't find the container with id d400cc687f0e01b92e31d60f4fc9c06ec873227f815ba94f9aa332f622ce0627 Jul 21 22:35:38.936788 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.936769 2555 scope.go:117] "RemoveContainer" containerID="98283c258c2dcde559e7363115fd0052a994a8dd094e19053a2a6878c9f19489" Jul 21 22:35:38.973681 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.973657 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" event={"ID":"31c90d98-c368-4d38-b998-2807c1d5b8a2","Type":"ContainerStarted","Data":"9af6ece48d2f1289bceee84fc18f89fad7b4efd997ab51715137cabe7af4362b"} Jul 21 22:35:38.973794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.973705 2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" event={"ID":"31c90d98-c368-4d38-b998-2807c1d5b8a2","Type":"ContainerStarted","Data":"d400cc687f0e01b92e31d60f4fc9c06ec873227f815ba94f9aa332f622ce0627"} Jul 21 22:35:38.973794 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.973728 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:38.990755 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:38.990700 2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" podStartSLOduration=0.990687957 podStartE2EDuration="990.687957ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 22:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:35:38.989642657 +0000 UTC m=+2820.840402489" watchObservedRunningTime="2026-07-21 22:35:38.990687957 +0000 UTC m=+2820.841448105" Jul 21 22:35:40.243341 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:40.243305 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w5vvr_18704d0c-cabe-4eaf-aa71-73584e07fcc6/dns/0.log" Jul 21 22:35:40.266189 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:40.266163 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w5vvr_18704d0c-cabe-4eaf-aa71-73584e07fcc6/kube-rbac-proxy/0.log" Jul 21 22:35:40.290636 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:40.290594 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxh8g_782704f8-becd-4a10-b12f-e9c865f7625f/dns-node-resolver/0.log" Jul 21 22:35:40.929837 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:40.929801 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68c56456cb-xcdbv_57046bca-034f-4450-aa04-d5c7572b71d3/registry/0.log" Jul 21 22:35:40.954385 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:40.954361 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6v4wq_6f536e43-bf21-4c0f-b058-cbdf14b56af9/node-ca/0.log" Jul 21 22:35:42.250110 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:42.250080 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4n2wk_366c5814-8054-4caf-bab0-c8633d3959e2/discovery/0.log" Jul 21 22:35:42.274134 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:42.274111 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-68vxd_5d01a020-ddca-4912-9446-3dd9e3691913/kube-auth-proxy/0.log" Jul 21 22:35:43.216483 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:43.216449 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d4cg9_1f12964a-4dd2-40d4-ae0e-53f722f5eaa1/serve-healthcheck-canary/0.log" Jul 21 22:35:43.847765 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:43.847737 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8dt8_b938b803-51a6-42c1-b29e-42c7aa84d2eb/kube-rbac-proxy/0.log" Jul 21 22:35:43.870032 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:43.870012 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8dt8_b938b803-51a6-42c1-b29e-42c7aa84d2eb/exporter/0.log" Jul 21 22:35:43.895396 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:43.895367 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8dt8_b938b803-51a6-42c1-b29e-42c7aa84d2eb/extractor/0.log" Jul 21 22:35:44.987356 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:44.987331 2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-z9f6r" Jul 21 22:35:46.431174 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:46.431136 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-f5tbw_1af52da3-84bd-4c78-8b5f-ce023a1a2554/manager/0.log" Jul 21 22:35:46.826204 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:46.826181 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-dgq64_f3b6e149-0a77-4fd9-8a81-ca2e4763c237/manager/0.log" Jul 21 22:35:46.851374 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:46.851350 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-dgq64_f3b6e149-0a77-4fd9-8a81-ca2e4763c237/copy-manifests/0.log" Jul 21 22:35:48.621858 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:48.621813 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4n2wk_366c5814-8054-4caf-bab0-c8633d3959e2/discovery/0.log" Jul 21 22:35:48.634756 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:48.634732 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-68vxd_5d01a020-ddca-4912-9446-3dd9e3691913/kube-auth-proxy/0.log" Jul 21 22:35:50.171951 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:50.171925 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4n2wk_366c5814-8054-4caf-bab0-c8633d3959e2/discovery/0.log" Jul 21 22:35:50.228270 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:50.228251 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-68vxd_5d01a020-ddca-4912-9446-3dd9e3691913/kube-auth-proxy/0.log" Jul 21 22:35:53.719874 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:53.719838 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2fjtw_cef6ce6a-368e-4309-8435-53804d212c25/migrator/0.log" Jul 21 22:35:53.753881 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:53.753861 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2fjtw_cef6ce6a-368e-4309-8435-53804d212c25/graceful-termination/0.log" Jul 21 22:35:55.300670 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.300639 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/kube-multus-additional-cni-plugins/0.log" Jul 21 22:35:55.330881 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.330858 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/egress-router-binary-copy/0.log" Jul 21 22:35:55.355187 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.355167 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/cni-plugins/0.log" Jul 21 22:35:55.381808 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.381794 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/bond-cni-plugin/0.log" Jul 21 22:35:55.412554 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.412538 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/routeoverride-cni/0.log" Jul 21 22:35:55.436157 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.436128 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/whereabouts-cni-bincopy/0.log" Jul 21 22:35:55.483395 ip-10-0-130-224 kubenswrapper[2555]: I0721 22:35:55.483370 2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2n7nf_6aebda04-dee6-4b4f-8434-721d2c1a5936/whereabouts-cni/0.log"