Jul 23 18:16:13.151505 ip-10-0-135-127 systemd[1]: Starting Kubernetes Kubelet... Jul 23 18:16:13.572552 ip-10-0-135-127 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:16:13.572552 ip-10-0-135-127 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 18:16:13.572552 ip-10-0-135-127 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:16:13.572552 ip-10-0-135-127 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 18:16:13.572552 ip-10-0-135-127 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.573184 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579762 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579787 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579792 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579795 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579799 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579802 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579806 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579808 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579811 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579814 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579817 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579820 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579823 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579826 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579831 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579843 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579846 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579848 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.652700 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579851 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579854 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579858 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579862 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579865 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579868 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579871 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579874 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579880 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579883 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579886 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579890 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579893 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579896 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579898 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579903 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579907 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579912 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579914 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.653343 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579917 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579923 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579926 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579930 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579934 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579938 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579949 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579956 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579959 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579962 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579964 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579973 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579976 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579982 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579985 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579988 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579991 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579994 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579996 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.654211 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.579999 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580002 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580004 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580007 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580010 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580013 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580019 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580022 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580025 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580028 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580031 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580033 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580036 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580039 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580042 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580044 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580047 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580050 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580055 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.654858 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580058 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580060 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580063 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580066 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580068 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580071 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580079 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580082 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580085 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580087 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580090 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580093 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580100 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580105 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580110 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580118 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580122 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580125 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580128 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580130 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.655498 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580133 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580136 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580139 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.580141 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582848 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582857 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582860 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582864 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582868 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582871 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582878 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582881 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582884 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582887 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582890 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582893 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582895 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582898 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582901 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582911 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.656146 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582914 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582917 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582920 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582925 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582929 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582932 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582935 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582938 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582940 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582943 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582946 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582950 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582954 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582957 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582960 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582968 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582971 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582976 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582980 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.656802 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582983 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582987 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582990 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.582995 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583000 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583005 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583009 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583012 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583017 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583020 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583022 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583025 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583028 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583037 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583040 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583043 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583046 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583048 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583051 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583054 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.657464 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583056 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583062 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583064 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583067 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583069 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583072 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583074 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583077 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583081 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583084 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583087 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583089 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583092 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583094 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583099 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583102 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583106 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583108 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583111 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583114 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.658120 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583116 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583119 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583123 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583127 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583130 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583135 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583144 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583147 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583151 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583155 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583159 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583164 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583168 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583171 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583174 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583177 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583179 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583182 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583187 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583190 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.658999 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583194 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583197 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583199 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583202 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583928 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583934 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583938 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583941 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583946 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583949 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583951 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583954 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583957 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583959 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583965 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583968 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583971 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583975 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.583996 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.660030 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584001 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584003 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584009 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584011 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584014 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584017 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584020 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584023 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584027 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584031 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584034 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584037 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584040 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584045 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584050 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584054 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584056 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584060 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584063 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.660564 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584065 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584068 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584071 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584074 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584077 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584080 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584083 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584086 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584093 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584096 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584099 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584102 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584105 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584113 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584117 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584120 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584123 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584125 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584130 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.661122 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584134 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584141 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584145 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584149 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584158 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584163 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584165 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584169 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584172 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584175 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584178 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584180 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584183 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584189 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584191 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584194 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584197 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584199 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584202 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584205 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.661755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584207 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.662197 ip-10-0-135-127 systemd[1]: Started Kubernetes Kubelet. Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584212 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584216 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584221 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584239 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584242 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584245 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584255 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584258 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584261 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584264 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584267 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584270 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584273 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584276 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584279 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584281 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584287 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584289 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584293 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.662771 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584296 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584298 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584302 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584306 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584310 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.584314 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584403 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584414 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584425 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584429 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584434 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584437 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584441 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584447 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584453 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584456 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584459 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584469 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584475 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584480 2576 flags.go:64] FLAG: --cgroup-root="" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584498 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584502 2576 flags.go:64] FLAG: --client-ca-file="" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584508 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584511 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 18:16:13.663657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584518 2576 flags.go:64] FLAG: --cluster-domain="" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584521 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584524 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584527 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584531 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584534 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584537 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584544 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584548 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584551 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584554 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584556 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584560 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584563 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584567 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584572 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584575 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584578 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584584 2576 flags.go:64] FLAG: --enable-server="true" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584587 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584594 2576 flags.go:64] FLAG: --event-burst="100" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584599 2576 flags.go:64] FLAG: --event-qps="50" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584602 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584606 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 18:16:13.664679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584611 2576 flags.go:64] FLAG: --eviction-hard="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584615 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584618 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584621 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584624 2576 flags.go:64] FLAG: --eviction-soft="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584633 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584639 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584644 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584659 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584663 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584666 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584669 2576 flags.go:64] FLAG: --feature-gates="" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584673 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584676 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584679 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584683 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584689 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584692 2576 flags.go:64] FLAG: --help="false" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584695 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584698 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584702 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584705 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584708 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 18:16:13.665582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584712 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584715 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584720 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584723 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584726 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584729 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584734 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584737 2576 flags.go:64] FLAG: --kube-reserved="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584740 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584743 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584748 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584751 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584754 2576 flags.go:64] FLAG: --lock-file="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584757 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584760 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584769 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584775 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584778 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584780 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584785 2576 flags.go:64] FLAG: --logging-format="text" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584789 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584792 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584795 2576 flags.go:64] FLAG: --manifest-url="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584798 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584806 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 18:16:13.666812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584812 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584824 2576 flags.go:64] FLAG: --max-pods="110" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584830 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584833 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584836 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584839 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584842 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584845 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584848 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584858 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584861 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584865 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584874 2576 flags.go:64] FLAG: --pod-cidr="" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584879 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584890 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584893 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584896 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584899 2576 flags.go:64] FLAG: --port="10250" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584905 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584908 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05839a8c67e34dab9" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584911 2576 flags.go:64] FLAG: --qos-reserved="" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584914 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584918 2576 flags.go:64] FLAG: --register-node="true" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584928 2576 flags.go:64] FLAG: --register-with-taints="" Jul 23 18:16:13.668067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584932 2576 flags.go:64] FLAG: --registry-burst="10" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584935 2576 flags.go:64] FLAG: --registry-qps="5" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584941 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584944 2576 flags.go:64] FLAG: --reserved-memory="" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584947 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584950 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584953 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584956 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584961 2576 flags.go:64] FLAG: --runonce="false" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584965 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584975 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584984 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584989 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584992 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.584995 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585000 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585005 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585010 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585147 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585156 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585160 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585164 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585167 2576 flags.go:64] FLAG: --system-cgroups="" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585171 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585182 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 18:16:13.668772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585186 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585189 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585194 2576 flags.go:64] FLAG: --tls-min-version="" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585197 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585200 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585203 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585206 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585209 2576 flags.go:64] FLAG: --v="2" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585214 2576 flags.go:64] FLAG: --version="false" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585218 2576 flags.go:64] FLAG: --vmodule="" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585234 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585238 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585471 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585476 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585480 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585482 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585485 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585489 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585492 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585494 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585497 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585499 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.746364 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585502 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585505 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585507 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585510 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585513 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585516 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585518 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585522 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585526 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585528 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585531 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585537 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585540 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585543 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585545 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585548 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585551 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585553 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585556 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585558 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.747089 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585561 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585563 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585565 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585568 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585571 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585573 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585576 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585579 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585582 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585584 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585587 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585589 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585591 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585594 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585596 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585599 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585602 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585604 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585607 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585609 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.747860 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585612 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585618 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585622 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585626 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585630 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585633 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585635 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585638 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585640 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585643 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585646 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585648 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585651 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585653 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585656 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585658 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585661 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585663 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585666 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585668 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.748627 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585671 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585674 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585678 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585680 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585683 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585687 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585689 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585692 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585699 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585701 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585704 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585706 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585709 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585712 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585715 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585719 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585722 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585725 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585728 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.749644 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585730 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585733 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585736 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585738 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585741 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585743 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585746 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585748 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585751 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.585753 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.585758 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.592156 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.592169 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.592912 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.592923 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 18:16:13.750286 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.592983 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.592988 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.592991 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.592995 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.592999 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593002 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593005 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593007 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593010 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593013 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593015 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593018 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593020 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593023 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593025 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593028 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593031 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593033 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593036 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593040 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.750781 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593043 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593045 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593048 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593051 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593053 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593056 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593058 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593061 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593063 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593066 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593069 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593072 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593075 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593078 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593081 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593083 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593086 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593088 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593091 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593093 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.751414 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593096 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593098 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593101 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593104 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593106 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593109 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593112 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593114 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593116 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593119 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593121 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593127 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593129 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593132 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593134 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593137 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593140 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593142 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593145 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.751950 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593149 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593153 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593156 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593160 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593162 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593166 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593169 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593171 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593174 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593176 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593179 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593182 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593184 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593187 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593189 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593192 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593194 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593197 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593200 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.752755 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593202 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593205 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593208 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593212 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593215 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593219 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593239 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593244 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593248 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593252 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593255 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593258 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593262 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593266 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593269 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593272 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593274 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593278 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593282 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.753404 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593285 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593287 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.593292 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593425 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593431 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593433 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593436 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593439 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593442 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593444 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593447 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593450 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593452 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593455 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593457 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:16:13.754045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593461 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593465 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593468 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593470 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593474 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593477 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593481 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593484 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593487 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593490 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593492 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593495 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593499 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593502 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593505 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593507 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593513 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593516 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593518 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:16:13.754481 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593521 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593523 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593526 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593528 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593531 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593533 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593536 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593539 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593541 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593544 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593547 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593549 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593552 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593554 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593557 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593559 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593562 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593566 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593568 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:16:13.755103 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593571 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593573 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593576 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593578 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593581 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593583 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593586 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593589 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593591 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593594 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593598 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593607 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593610 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593612 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593615 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593617 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593620 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593622 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593625 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593627 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:16:13.755730 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593629 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593632 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593635 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593637 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593640 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593642 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593645 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593647 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593650 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593653 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593656 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593659 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593661 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593664 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593667 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593669 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593672 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593674 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593677 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:16:13.756354 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593679 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593682 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593684 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593691 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593693 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593702 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593704 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593707 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593709 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:13.593712 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.593717 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.593804 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.595750 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.597021 2576 server.go:1019] "Starting client certificate rotation" Jul 23 18:16:13.756978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.597127 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.597166 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.599862 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.601211 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.619923 2576 log.go:25] "Validated CRI v1 runtime API" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.622245 2576 log.go:25] "Validated CRI v1 image API" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.623845 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.626186 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ecd92791-ac43-41c6-bbe9-3c4054ca6eab:/dev/nvme0n1p4 fa1242db-5d3f-4448-9d1a-bd367ad5ea88:/dev/nvme0n1p3] Jul 23 18:16:13.757600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.626199 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.640945 2576 manager.go:216] Machine: {Timestamp:2026-07-23 18:16:13.639157139 +0000 UTC m=+0.369832221 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100013 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0928ba851017c30889ccadf23eec SystemUUID:ec2e0928-ba85-1017-c308-89ccadf23eec BootID:11deac34-b87f-44be-8338-13ae4393bc7b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:78:6b:ef:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:78:6b:ef:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:e2:1c:91:62:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.641034 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.641125 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.642514 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.644042 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.644066 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.644839 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.644849 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.644856 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.645595 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.646916 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649655 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649767 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649794 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649807 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649817 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 18:16:13.758244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649826 2576 kubelet.go:398] "Adding apiserver pod source" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.649834 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.650127 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.650144 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.653561 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.655192 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.655219 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656246 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656258 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656267 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656273 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656281 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656286 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656293 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656298 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656305 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656310 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656318 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.656323 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.657464 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.657677 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.661410 2576 server.go:1268] "Started kubelet" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.661483 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.661616 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.661752 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.661690 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2n7d" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.663274 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.664336 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.664378 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.664450 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.664867 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.667373 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.669788 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.669799 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.669853 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670345 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670347 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670368 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670435 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670443 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 23 18:16:13.843221 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.670552 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2n7d" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.669647 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-127.ec2.internal.18c4fdf34757f54a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-127.ec2.internal,UID:ip-10-0-135-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-127.ec2.internal,},FirstTimestamp:2026-07-23 18:16:13.661386058 +0000 UTC m=+0.392061141,LastTimestamp:2026-07-23 18:16:13.661386058 +0000 UTC m=+0.392061141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-127.ec2.internal,}" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672017 2576 factory.go:55] Registering systemd factory Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672030 2576 factory.go:223] Registration of the systemd container factory successfully Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672369 2576 factory.go:153] Registering CRI-O factory Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672379 2576 factory.go:223] Registration of the crio container factory successfully Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672421 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672443 2576 factory.go:103] Registering Raw factory Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672453 2576 manager.go:1196] Started watching for new ooms in manager Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.672732 2576 manager.go:349] Starting recovery of all containers Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.677910 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.681973 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-127.ec2.internal\" not found" node="ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.682427 2576 manager.go:354] Recovery completed Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.695400 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.697679 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.697701 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.697711 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698178 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698185 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698196 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698202 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698866 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698895 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.698906 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.699312 2576 policy_none.go:49] "None policy: Start" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.699326 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.699335 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.700282 2576 policy_none.go:47] "Start" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.737922 2576 manager.go:337] "Starting Device Plugin manager" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.737971 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.737983 2576 server.go:87] "Starting device plugin registration server" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.738192 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.738203 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.738518 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.738582 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.738588 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.739091 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.739216 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.826915 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.828056 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.828086 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.828103 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.828135 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.830648 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:13.859515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.838810 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.840352 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.840371 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.840380 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.840415 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.847069 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.847084 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-127.ec2.internal\": node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:13.866634 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.863684 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:13.931255 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.929073 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal"] Jul 23 18:16:13.931255 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.929118 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.931255 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.930470 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.931255 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.930485 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.931255 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.930496 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.931662 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.931811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.931838 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932425 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932444 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932452 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932760 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932779 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.932791 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.933499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.933524 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.934204 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.934240 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:16:13.938249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:13.934253 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:16:14.036996 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.964209 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.036996 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.978782 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-127.ec2.internal\" not found" node="ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.036996 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:13.983417 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-127.ec2.internal\" not found" node="ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.064722 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.071829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dd7017dd1e272c050bc3c493997c4c9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-127.ec2.internal\" (UID: \"8dd7017dd1e272c050bc3c493997c4c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.071855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.071881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.165506 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dd7017dd1e272c050bc3c493997c4c9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-127.ec2.internal\" (UID: \"8dd7017dd1e272c050bc3c493997c4c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dd7017dd1e272c050bc3c493997c4c9-config\") pod \"kube-apiserver-proxy-ip-10-0-135-127.ec2.internal\" (UID: \"8dd7017dd1e272c050bc3c493997c4c9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.197981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.172486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04b86fa819b797170650f264b6143415-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal\" (UID: \"04b86fa819b797170650f264b6143415\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.317831 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.273152 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.317831 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.282011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.317831 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.287125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.451742 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.373801 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.474664 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-127.ec2.internal\" not found" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.537745 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.572304 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.581740 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.583686 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.595403 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.596630 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.596727 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.596723 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:16:14.613063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.596723 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.653967 2576 apiserver.go:52] "Watching apiserver" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.670740 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.671507 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.674542 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vrf52","openshift-multus/multus-zw522","openshift-multus/network-metrics-daemon-wc7ll","openshift-network-diagnostics/network-check-target-g7glb","openshift-network-operator/iptables-alerter-p749r","kube-system/global-pull-secret-syncer-wdlln","kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch","openshift-dns/node-resolver-6bnvn","openshift-image-registry/node-ca-dcl42","openshift-ovn-kubernetes/ovnkube-node-xcgwc","kube-system/konnectivity-agent-7ppq8","openshift-cluster-node-tuning-operator/tuned-wbfzv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal"] Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.675269 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 18:11:14 +0000 UTC" deadline="2028-01-05 00:20:01.115405976 +0000 UTC" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.675304 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12726h3m46.440105087s" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.676204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.677975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.678545 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.678569 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.678605 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lv6zs\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.678644 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.679165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.679262 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.680058 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.680549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.680611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.680837 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.680873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6vgps\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.681110 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.681813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.682956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.683006 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.683711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.683917 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.684027 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.684119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hspk7\"" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.684136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.744744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.685362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.686190 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.686346 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4lsqb\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.686378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.686802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zw522" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.687452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.687836 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.687880 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.688137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.688217 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4g4k\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.688991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.689182 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s7skp\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.689207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.689377 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.689420 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.689604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.690118 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.690133 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.690199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xp5hv\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.691459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.691619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.691750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sr27z\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.691787 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.691948 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.692259 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.692835 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.694677 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.702073 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.702151 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k72gt\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.702284 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.704267 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqpxl" Jul 23 18:16:14.746909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.712177 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sqpxl" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.771354 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-etc-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-script-lib\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-conf\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-tmp\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-os-release\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-multus\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-system-cni-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-agent-certs\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysconfig\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-system-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.776981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-ovn\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-node-log\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.860741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zhx\" (UniqueName: \"kubernetes.io/projected/17b9081b-b362-48a0-8fb0-c53eaf57438e-kube-api-access-f5zhx\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f169b268-cb81-47b7-a6ae-b4305a26c317-hosts-file\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-netns\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-cnibin\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-os-release\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-dbus\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-socket-dir-parent\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777203 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-registration-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-netd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-kubernetes\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-bin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777343 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b268-cb81-47b7-a6ae-b4305a26c317-tmp-dir\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-sys-fs\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.861526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-modprobe-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-host\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-tuned\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpjv7\" (UniqueName: \"kubernetes.io/projected/a4ec536d-8503-4045-bae0-908bd6c9f99a-kube-api-access-rpjv7\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-konnectivity-ca\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-systemd\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-run\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-slash\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-netns\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-env-overrides\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovn-node-metrics-cert\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-lib-modules\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-var-lib-kubelet\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-cnibin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-cni-binary-copy\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-hostroot\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-log-socket\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hfq\" (UniqueName: \"kubernetes.io/projected/3b022432-5ff4-48d8-8935-a5d79acbccbd-kube-api-access-n5hfq\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-device-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-sys\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnfq\" (UniqueName: \"kubernetes.io/projected/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-kube-api-access-qlnfq\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-kubelet\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-kubelet-config\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-socket-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.777982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b022432-5ff4-48d8-8935-a5d79acbccbd-iptables-alerter-script\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-k8s-cni-cncf-io\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-conf-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.862839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-multus-certs\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwvf\" (UniqueName: \"kubernetes.io/projected/6201bed0-3725-46e8-aa9a-25259f731d44-kube-api-access-hjwvf\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b022432-5ff4-48d8-8935-a5d79acbccbd-host-slash\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtm9\" (UniqueName: \"kubernetes.io/projected/f169b268-cb81-47b7-a6ae-b4305a26c317-kube-api-access-vjtm9\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6201bed0-3725-46e8-aa9a-25259f731d44-serviceca\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-systemd-units\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-systemd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6201bed0-3725-46e8-aa9a-25259f731d44-host\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsn2x\" (UniqueName: \"kubernetes.io/projected/17732286-b41e-4c5d-913a-440a7677d70d-kube-api-access-lsn2x\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-kubelet\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-etc-kubernetes\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-var-lib-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-bin\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-config\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.863484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rfh\" (UniqueName: \"kubernetes.io/projected/ca54f61e-43d9-4fef-b083-44ca41a44534-kube-api-access-c4rfh\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.864218 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-daemon-config\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.864218 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.778507 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p45v\" (UniqueName: \"kubernetes.io/projected/4952ba4f-813b-43db-b65f-abba33ad4d2a-kube-api-access-4p45v\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.878851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-kubernetes\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.878896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-bin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.878934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.878959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b268-cb81-47b7-a6ae-b4305a26c317-tmp-dir\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-kubernetes\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-bin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-sys-fs\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.879061 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-modprobe-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-host\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.879127 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:15.379105244 +0000 UTC m=+2.109780325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879147 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-host\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-tuned\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpjv7\" (UniqueName: \"kubernetes.io/projected/a4ec536d-8503-4045-bae0-908bd6c9f99a-kube-api-access-rpjv7\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-sys-fs\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-konnectivity-ca\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.954773 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-systemd\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b268-cb81-47b7-a6ae-b4305a26c317-tmp-dir\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-run\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-slash\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-slash\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-modprobe-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-run\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.879943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-systemd\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-netns\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-env-overrides\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovn-node-metrics-cert\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-lib-modules\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-var-lib-kubelet\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-cnibin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-cni-binary-copy\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-hostroot\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.958289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-log-socket\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hfq\" (UniqueName: \"kubernetes.io/projected/3b022432-5ff4-48d8-8935-a5d79acbccbd-kube-api-access-n5hfq\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-device-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-sys\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-konnectivity-ca\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnfq\" (UniqueName: \"kubernetes.io/projected/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-kube-api-access-qlnfq\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880834 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-kubelet\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-kubelet-config\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.880982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-socket-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881027 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-env-overrides\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b022432-5ff4-48d8-8935-a5d79acbccbd-iptables-alerter-script\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-netns\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.959084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-k8s-cni-cncf-io\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-conf-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-multus-certs\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwvf\" (UniqueName: \"kubernetes.io/projected/6201bed0-3725-46e8-aa9a-25259f731d44-kube-api-access-hjwvf\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-hostroot\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-kubelet\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-socket-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-k8s-cni-cncf-io\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b022432-5ff4-48d8-8935-a5d79acbccbd-host-slash\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.881943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-conf-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.882036 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-log-socket\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.882129 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:15.382075229 +0000 UTC m=+2.112750304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882190 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.960020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtm9\" (UniqueName: \"kubernetes.io/projected/f169b268-cb81-47b7-a6ae-b4305a26c317-kube-api-access-vjtm9\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6201bed0-3725-46e8-aa9a-25259f731d44-serviceca\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b022432-5ff4-48d8-8935-a5d79acbccbd-iptables-alerter-script\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-device-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-systemd-units\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-d\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-systemd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882535 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-systemd-units\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6201bed0-3725-46e8-aa9a-25259f731d44-host\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-multus-certs\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-sys\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6201bed0-3725-46e8-aa9a-25259f731d44-host\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b022432-5ff4-48d8-8935-a5d79acbccbd-host-slash\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-var-lib-kubelet\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.960734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-systemd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-lib-modules\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-cnibin\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6201bed0-3725-46e8-aa9a-25259f731d44-serviceca\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.882997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsn2x\" (UniqueName: \"kubernetes.io/projected/17732286-b41e-4c5d-913a-440a7677d70d-kube-api-access-lsn2x\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-kubelet\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-kubelet-config\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-etc-kubernetes\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-var-lib-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-bin\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-config\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rfh\" (UniqueName: \"kubernetes.io/projected/ca54f61e-43d9-4fef-b083-44ca41a44534-kube-api-access-c4rfh\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883420 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-daemon-config\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p45v\" (UniqueName: \"kubernetes.io/projected/4952ba4f-813b-43db-b65f-abba33ad4d2a-kube-api-access-4p45v\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-etc-kubernetes\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-cni-binary-copy\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.961762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-etc-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-script-lib\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-conf\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-tmp\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-os-release\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-multus\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-bin\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-system-cni-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-agent-certs\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-kubelet\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysconfig\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-system-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-ovn\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-node-log\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zhx\" (UniqueName: \"kubernetes.io/projected/17b9081b-b362-48a0-8fb0-c53eaf57438e-kube-api-access-f5zhx\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.962517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f169b268-cb81-47b7-a6ae-b4305a26c317-hosts-file\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-netns\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-cnibin\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-os-release\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-dbus\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-daemon-config\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-socket-dir-parent\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-node-log\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-registration-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-system-cni-dir\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-netd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-host-cni-netd\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-etc-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-config\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.963504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b31242ed-e9d8-4a85-b061-2b226f509c93-dbus\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-multus-socket-dir-parent\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ec536d-8503-4045-bae0-908bd6c9f99a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f169b268-cb81-47b7-a6ae-b4305a26c317-hosts-file\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.884962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-registration-dir\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-system-cni-dir\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-run-netns\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-ovn\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-cnibin\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysctl-conf\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca54f61e-43d9-4fef-b083-44ca41a44534-etc-selinux\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a4ec536d-8503-4045-bae0-908bd6c9f99a-os-release\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-host-var-lib-cni-multus\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-sysconfig\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4952ba4f-813b-43db-b65f-abba33ad4d2a-os-release\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885455 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-run-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.883891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17b9081b-b362-48a0-8fb0-c53eaf57438e-var-lib-openvswitch\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.885881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovnkube-script-lib\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.964254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.887310 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17b9081b-b362-48a0-8fb0-c53eaf57438e-ovn-node-metrics-cert\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.887429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b59cdc50-ae0b-4b8e-8741-0dc971d29f91-agent-certs\") pod \"konnectivity-agent-7ppq8\" (UID: \"b59cdc50-ae0b-4b8e-8741-0dc971d29f91\") " pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.887693 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-tmp\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.897714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-etc-tuned\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.897885 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.897907 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.897947 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:14.898012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:15.397994132 +0000 UTC m=+2.128669224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsn2x\" (UniqueName: \"kubernetes.io/projected/17732286-b41e-4c5d-913a-440a7677d70d-kube-api-access-lsn2x\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpjv7\" (UniqueName: \"kubernetes.io/projected/a4ec536d-8503-4045-bae0-908bd6c9f99a-kube-api-access-rpjv7\") pod \"multus-additional-cni-plugins-vrf52\" (UID: \"a4ec536d-8503-4045-bae0-908bd6c9f99a\") " pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnfq\" (UniqueName: \"kubernetes.io/projected/ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c-kube-api-access-qlnfq\") pod \"tuned-wbfzv\" (UID: \"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c\") " pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rfh\" (UniqueName: \"kubernetes.io/projected/ca54f61e-43d9-4fef-b083-44ca41a44534-kube-api-access-c4rfh\") pod \"aws-ebs-csi-driver-node-qs7ch\" (UID: \"ca54f61e-43d9-4fef-b083-44ca41a44534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwvf\" (UniqueName: \"kubernetes.io/projected/6201bed0-3725-46e8-aa9a-25259f731d44-kube-api-access-hjwvf\") pod \"node-ca-dcl42\" (UID: \"6201bed0-3725-46e8-aa9a-25259f731d44\") " pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.898578 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtm9\" (UniqueName: \"kubernetes.io/projected/f169b268-cb81-47b7-a6ae-b4305a26c317-kube-api-access-vjtm9\") pod \"node-resolver-6bnvn\" (UID: \"f169b268-cb81-47b7-a6ae-b4305a26c317\") " pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.900368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hfq\" (UniqueName: \"kubernetes.io/projected/3b022432-5ff4-48d8-8935-a5d79acbccbd-kube-api-access-n5hfq\") pod \"iptables-alerter-p749r\" (UID: \"3b022432-5ff4-48d8-8935-a5d79acbccbd\") " pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.900957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zhx\" (UniqueName: \"kubernetes.io/projected/17b9081b-b362-48a0-8fb0-c53eaf57438e-kube-api-access-f5zhx\") pod \"ovnkube-node-xcgwc\" (UID: \"17b9081b-b362-48a0-8fb0-c53eaf57438e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.901019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p45v\" (UniqueName: \"kubernetes.io/projected/4952ba4f-813b-43db-b65f-abba33ad4d2a-kube-api-access-4p45v\") pod \"multus-zw522\" (UID: \"4952ba4f-813b-43db-b65f-abba33ad4d2a\") " pod="openshift-multus/multus-zw522" Jul 23 18:16:14.965185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:14.909540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:15.054932 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.035103 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcl42" Jul 23 18:16:15.060042 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.058939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.083249 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p749r" Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.094750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bnvn" Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.116317 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf169b268_cb81_47b7_a6ae_b4305a26c317.slice/crio-cbe96422e2293affa165861797077978b6819f0939ad4c419f4a80e297a0ac42 WatchSource:0}: Error finding container cbe96422e2293affa165861797077978b6819f0939ad4c419f4a80e297a0ac42: Status 404 returned error can't find the container with id cbe96422e2293affa165861797077978b6819f0939ad4c419f4a80e297a0ac42 Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.117276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b86fa819b797170650f264b6143415.slice/crio-aa547d96954f775acb9c5b8a8f30f2a5a66830b52918d8f41e5ca10ca8047adc WatchSource:0}: Error finding container aa547d96954f775acb9c5b8a8f30f2a5a66830b52918d8f41e5ca10ca8047adc: Status 404 returned error can't find the container with id aa547d96954f775acb9c5b8a8f30f2a5a66830b52918d8f41e5ca10ca8047adc Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.117651 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b022432_5ff4_48d8_8935_a5d79acbccbd.slice/crio-e4a015713d50879402285fa9b422422c6ed25ab361058c8868a78edd6071f8c0 WatchSource:0}: Error finding container e4a015713d50879402285fa9b422422c6ed25ab361058c8868a78edd6071f8c0: Status 404 returned error can't find the container with id e4a015713d50879402285fa9b422422c6ed25ab361058c8868a78edd6071f8c0 Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.118992 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd7017dd1e272c050bc3c493997c4c9.slice/crio-431525f49da5a0390dc65c1520f4bf7a4443a7c310c1e7b0284a200e2e908df4 WatchSource:0}: Error finding container 431525f49da5a0390dc65c1520f4bf7a4443a7c310c1e7b0284a200e2e908df4: Status 404 returned error can't find the container with id 431525f49da5a0390dc65c1520f4bf7a4443a7c310c1e7b0284a200e2e908df4 Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.122504 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.133396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.144945 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca54f61e_43d9_4fef_b083_44ca41a44534.slice/crio-7ba8653cf22bb565994c654f1153f11229beac20ada361ce490478d6e3960571 WatchSource:0}: Error finding container 7ba8653cf22bb565994c654f1153f11229beac20ada361ce490478d6e3960571: Status 404 returned error can't find the container with id 7ba8653cf22bb565994c654f1153f11229beac20ada361ce490478d6e3960571 Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.153020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zw522" Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.165877 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4952ba4f_813b_43db_b65f_abba33ad4d2a.slice/crio-1c1580d0fe503153c6e2021a9c2c159cb9d181d24494f0c0c0f8b9af55e5ca12 WatchSource:0}: Error finding container 1c1580d0fe503153c6e2021a9c2c159cb9d181d24494f0c0c0f8b9af55e5ca12: Status 404 returned error can't find the container with id 1c1580d0fe503153c6e2021a9c2c159cb9d181d24494f0c0c0f8b9af55e5ca12 Jul 23 18:16:15.181520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.176542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrf52" Jul 23 18:16:15.335071 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.184543 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ec536d_8503_4045_bae0_908bd6c9f99a.slice/crio-79d71af00ff6d3e648dbaff56973e2513c9387c3b5e77ce289b4df16d16eb6e8 WatchSource:0}: Error finding container 79d71af00ff6d3e648dbaff56973e2513c9387c3b5e77ce289b4df16d16eb6e8: Status 404 returned error can't find the container with id 79d71af00ff6d3e648dbaff56973e2513c9387c3b5e77ce289b4df16d16eb6e8 Jul 23 18:16:15.335071 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.187651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:15.335071 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.194959 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b9081b_b362_48a0_8fb0_c53eaf57438e.slice/crio-02917abf986861d705f484c33b996a2070ec30942bdafdfede780e656b9ffabb WatchSource:0}: Error finding container 02917abf986861d705f484c33b996a2070ec30942bdafdfede780e656b9ffabb: Status 404 returned error can't find the container with id 02917abf986861d705f484c33b996a2070ec30942bdafdfede780e656b9ffabb Jul 23 18:16:15.335071 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.246084 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:15.335071 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.297385 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59cdc50_ae0b_4b8e_8741_0dc971d29f91.slice/crio-0725509b8651767d91fbc9e59ac9c65ecce318518f4e34c1a6b20045562af25c WatchSource:0}: Error finding container 0725509b8651767d91fbc9e59ac9c65ecce318518f4e34c1a6b20045562af25c: Status 404 returned error can't find the container with id 0725509b8651767d91fbc9e59ac9c65ecce318518f4e34c1a6b20045562af25c Jul 23 18:16:15.388349 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.388314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:15.388476 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.388426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:15.388476 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.388445 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:15.388593 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.388485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:16.38847156 +0000 UTC m=+3.119146630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:15.388593 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.388551 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:15.388687 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.388610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:16.388592488 +0000 UTC m=+3.119267557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:15.420657 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.420640 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac66d55c_16b8_4e3e_ab69_a5fea9e1bb0c.slice/crio-a63b0494ca29002185d172d4664c5872ba866a704f2e1f5d802e295410719508 WatchSource:0}: Error finding container a63b0494ca29002185d172d4664c5872ba866a704f2e1f5d802e295410719508: Status 404 returned error can't find the container with id a63b0494ca29002185d172d4664c5872ba866a704f2e1f5d802e295410719508 Jul 23 18:16:15.489483 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.489437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:15.489573 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.489530 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:15.489573 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.489542 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:15.489573 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.489550 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:15.489684 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:15.489588 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:16.489576933 +0000 UTC m=+3.220252003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:15.596677 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.596225 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:15.620749 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:15.620723 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6201bed0_3725_46e8_aa9a_25259f731d44.slice/crio-c398c084030eba6cd09ee44151b417d77486054b11a3023963bb332464a336c7 WatchSource:0}: Error finding container c398c084030eba6cd09ee44151b417d77486054b11a3023963bb332464a336c7: Status 404 returned error can't find the container with id c398c084030eba6cd09ee44151b417d77486054b11a3023963bb332464a336c7 Jul 23 18:16:15.714352 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.714294 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:11:15 +0000 UTC" deadline="2028-04-06 13:37:33.8223329 +0000 UTC" Jul 23 18:16:15.714352 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.714344 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14947h21m18.1079945s" Jul 23 18:16:15.792855 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.792590 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:15.851479 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.851418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"02917abf986861d705f484c33b996a2070ec30942bdafdfede780e656b9ffabb"} Jul 23 18:16:15.858114 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.858022 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" event={"ID":"ca54f61e-43d9-4fef-b083-44ca41a44534","Type":"ContainerStarted","Data":"7ba8653cf22bb565994c654f1153f11229beac20ada361ce490478d6e3960571"} Jul 23 18:16:15.867475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.867396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bnvn" event={"ID":"f169b268-cb81-47b7-a6ae-b4305a26c317","Type":"ContainerStarted","Data":"cbe96422e2293affa165861797077978b6819f0939ad4c419f4a80e297a0ac42"} Jul 23 18:16:15.881872 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.881835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p749r" event={"ID":"3b022432-5ff4-48d8-8935-a5d79acbccbd","Type":"ContainerStarted","Data":"e4a015713d50879402285fa9b422422c6ed25ab361058c8868a78edd6071f8c0"} Jul 23 18:16:15.892621 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.892590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" event={"ID":"04b86fa819b797170650f264b6143415","Type":"ContainerStarted","Data":"aa547d96954f775acb9c5b8a8f30f2a5a66830b52918d8f41e5ca10ca8047adc"} Jul 23 18:16:15.908177 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.908151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" event={"ID":"8dd7017dd1e272c050bc3c493997c4c9","Type":"ContainerStarted","Data":"431525f49da5a0390dc65c1520f4bf7a4443a7c310c1e7b0284a200e2e908df4"} Jul 23 18:16:15.934034 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.933992 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcl42" event={"ID":"6201bed0-3725-46e8-aa9a-25259f731d44","Type":"ContainerStarted","Data":"c398c084030eba6cd09ee44151b417d77486054b11a3023963bb332464a336c7"} Jul 23 18:16:15.940172 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.940145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" event={"ID":"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c","Type":"ContainerStarted","Data":"a63b0494ca29002185d172d4664c5872ba866a704f2e1f5d802e295410719508"} Jul 23 18:16:15.955926 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.955892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ppq8" event={"ID":"b59cdc50-ae0b-4b8e-8741-0dc971d29f91","Type":"ContainerStarted","Data":"0725509b8651767d91fbc9e59ac9c65ecce318518f4e34c1a6b20045562af25c"} Jul 23 18:16:15.986389 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:15.986357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerStarted","Data":"79d71af00ff6d3e648dbaff56973e2513c9387c3b5e77ce289b4df16d16eb6e8"} Jul 23 18:16:16.007542 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.007430 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zw522" event={"ID":"4952ba4f-813b-43db-b65f-abba33ad4d2a","Type":"ContainerStarted","Data":"1c1580d0fe503153c6e2021a9c2c159cb9d181d24494f0c0c0f8b9af55e5ca12"} Jul 23 18:16:16.396192 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.396150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:16.396399 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.396241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:16.396399 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.396394 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:16.396529 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.396458 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:18.396438007 +0000 UTC m=+5.127113081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:16.396884 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.396864 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:16.396978 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.396919 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:18.396903416 +0000 UTC m=+5.127578493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:16.497843 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.497230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:16.497843 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.497452 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:16.497843 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.497476 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:16.497843 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.497490 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:16.497843 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.497542 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:18.497524512 +0000 UTC m=+5.228199601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:16.715449 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.715339 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:11:15 +0000 UTC" deadline="2027-12-21 07:54:29.705767672 +0000 UTC" Jul 23 18:16:16.715449 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.715402 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12373h38m12.990369895s" Jul 23 18:16:16.828977 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.828894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:16.829168 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.829033 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:16.829918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.829281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:16.829918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:16.829352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:16.829918 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.829449 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:16.829918 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:16.829549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:17.711965 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:17.711759 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:16:18.410388 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.410299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:18.410388 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.410383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:18.410929 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.410567 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:18.410929 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.410627 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:22.410608229 +0000 UTC m=+9.141283306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:18.411060 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.411037 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:18.411129 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.411095 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:22.411075982 +0000 UTC m=+9.141751053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:18.510834 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.510796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:18.511020 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.510972 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:18.511020 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.510994 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:18.511020 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.511007 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:18.511187 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.511071 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:22.511051582 +0000 UTC m=+9.241726674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:18.828854 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.828761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:18.829092 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.828970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:18.829666 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.829392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:18.829666 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.829511 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:18.829666 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:18.829560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:18.829666 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:18.829629 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:20.829649 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:20.829570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:20.829649 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:20.829598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:20.830215 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:20.829579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:20.830215 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:20.829714 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:20.830215 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:20.829876 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:20.830215 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:20.830008 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:22.444277 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.444159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:22.444277 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.444253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:22.444811 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.444367 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:22.444811 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.444372 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:22.444811 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.444446 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:30.444425116 +0000 UTC m=+17.175100187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:22.444811 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.444469 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:30.444458097 +0000 UTC m=+17.175133191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:22.544608 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.544570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:22.544796 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.544716 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:22.544796 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.544741 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:22.544796 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.544755 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:22.544953 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.544816 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:30.544797385 +0000 UTC m=+17.275472468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:22.829423 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.829266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:22.829423 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.829387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:22.829423 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.829413 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:22.829733 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:22.829427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:22.829733 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.829510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:22.829733 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:22.829587 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:24.829113 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:24.829081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:24.829595 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:24.829201 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:24.829595 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:24.829083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:24.829595 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:24.829320 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:24.829595 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:24.829081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:24.829595 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:24.829413 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:26.828758 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:26.828725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:26.829165 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:26.828829 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:26.829165 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:26.828723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:26.829165 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:26.828735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:26.829165 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:26.828951 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:26.829165 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:26.829036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:28.828373 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:28.828346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:28.828766 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:28.828351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:28.828766 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:28.828351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:28.828766 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:28.828539 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:28.828766 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:28.828443 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:28.828766 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:28.828625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:30.504704 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.504668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:30.505142 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.504735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:30.505142 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.504840 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:30.505142 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.504897 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:46.50487916 +0000 UTC m=+33.235554248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:30.505142 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.504841 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:30.505142 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.504989 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:16:46.504972964 +0000 UTC m=+33.235648042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:30.605793 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.605760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:30.605941 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.605921 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:30.605941 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.605939 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:30.606060 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.605949 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:30.606060 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.605995 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:46.605978764 +0000 UTC m=+33.336653836 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:30.829091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.829022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:30.829091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.829051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:30.829326 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:30.829144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:30.829326 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.829143 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:30.829326 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.829294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:30.829455 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:30.829380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:32.828905 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:32.828852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:32.829390 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:32.828978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:32.829390 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:32.828992 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:32.829390 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:32.829014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:32.829390 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:32.829122 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:32.829390 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:32.829218 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:34.828838 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:34.828798 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:34.829278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:34.828918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:34.829278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:34.828946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:34.829278 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:34.828922 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:34.829278 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:34.829040 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:34.829278 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:34.829144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:36.829312 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:36.829158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:36.829754 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:36.829200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:36.829754 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:36.829409 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:36.829754 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:36.829457 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:36.829754 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:36.829247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:36.829754 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:36.829554 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:37.058549 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.058521 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="ce25a0a777cde5e97790011b2d5983252caf6226d0087052ab0e10f1ae4fa661" exitCode=0 Jul 23 18:16:37.058650 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.058617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"ce25a0a777cde5e97790011b2d5983252caf6226d0087052ab0e10f1ae4fa661"} Jul 23 18:16:37.060173 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.060146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zw522" event={"ID":"4952ba4f-813b-43db-b65f-abba33ad4d2a","Type":"ContainerStarted","Data":"2317f1dabcab67d9e7355505ab5093a3225c25cf070e82b688dc26ce489052fe"} Jul 23 18:16:37.062875 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.062858 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:16:37.063176 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063157 2576 generic.go:356] "Generic (PLEG): container finished" podID="17b9081b-b362-48a0-8fb0-c53eaf57438e" containerID="e7e3c91f57e26aa1abd08653a17fca30ce6d17a836f606b1ced2389b2758a8c4" exitCode=1 Jul 23 18:16:37.063284 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"1377cfb30106cd601a10b25133bf18f8a7fc8edaeda88ae004013d832256e421"} Jul 23 18:16:37.063337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"648a3bc9eee18b7d1c8d00209eeef4b70603aa6f94313b9da3bf7d33b47294dd"} Jul 23 18:16:37.063388 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"172d6ce264a093a3c90796cc6abaf59169666ef76d818db592b4a9f5b30af584"} Jul 23 18:16:37.063438 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"4fdd63017f92696d6642b5708bf488ad12528a0a934bf7c2f0d0807a748ca548"} Jul 23 18:16:37.063488 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerDied","Data":"e7e3c91f57e26aa1abd08653a17fca30ce6d17a836f606b1ced2389b2758a8c4"} Jul 23 18:16:37.063541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.063510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"853e6b67d93035c827c1bfe3034a8a5e70627bf8ad8f349afe71eab7ee4aeccc"} Jul 23 18:16:37.064753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.064728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" event={"ID":"ca54f61e-43d9-4fef-b083-44ca41a44534","Type":"ContainerStarted","Data":"a9e597860f3aae00f1e68d21e6fb0497a6c7b7a8b815cc999f096bf959fb9e7f"} Jul 23 18:16:37.066059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.066040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bnvn" event={"ID":"f169b268-cb81-47b7-a6ae-b4305a26c317","Type":"ContainerStarted","Data":"f1fa2da20f00f445fe8c454c0374f3c6614e18c2e08ab57e5a8940d4a66c20a8"} Jul 23 18:16:37.067473 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.067450 2576 generic.go:356] "Generic (PLEG): container finished" podID="04b86fa819b797170650f264b6143415" containerID="770c1d1f234736a1fe8d02f1d1549213322a506af6fc5ce37f0a8045e15cf7db" exitCode=0 Jul 23 18:16:37.067564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.067529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" event={"ID":"04b86fa819b797170650f264b6143415","Type":"ContainerDied","Data":"770c1d1f234736a1fe8d02f1d1549213322a506af6fc5ce37f0a8045e15cf7db"} Jul 23 18:16:37.068958 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.068929 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" event={"ID":"8dd7017dd1e272c050bc3c493997c4c9","Type":"ContainerStarted","Data":"ad95b81f6738db2441bc9d247ed4eff545c10723e0ce3c1892ca68f439717bba"} Jul 23 18:16:37.070730 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.070707 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcl42" event={"ID":"6201bed0-3725-46e8-aa9a-25259f731d44","Type":"ContainerStarted","Data":"3dbb0c708d2a78cfd67b73ee61ed8562bebfcbe6baa9d75b701b3fa50c1bba3f"} Jul 23 18:16:37.072243 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.072202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" event={"ID":"ac66d55c-16b8-4e3e-ab69-a5fea9e1bb0c","Type":"ContainerStarted","Data":"c97a421c9af54c89776d2eaf3f2ffc455e695468d6f069c7fe941530c2198c42"} Jul 23 18:16:37.073521 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.073495 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ppq8" event={"ID":"b59cdc50-ae0b-4b8e-8741-0dc971d29f91","Type":"ContainerStarted","Data":"023f2657b38d0e92585bd5b8c41eede56364dcbe3cd96130089232a9b4aa9dac"} Jul 23 18:16:37.110902 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.110667 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-127.ec2.internal" podStartSLOduration=22.110651881 podStartE2EDuration="22.110651881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.0971511 +0000 UTC m=+23.827826204" watchObservedRunningTime="2026-07-23 18:16:37.110651881 +0000 UTC m=+23.841326974" Jul 23 18:16:37.124119 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.124074 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dcl42" podStartSLOduration=2.756394111 podStartE2EDuration="23.124057422s" totalImagesPullingTime="20.367663311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.123644737 +0000 UTC m=+23.854319865" watchObservedRunningTime="2026-07-23 18:16:37.124057422 +0000 UTC m=+23.854732514" Jul 23 18:16:37.139403 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.139361 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wbfzv" podStartSLOduration=2.549189673 podStartE2EDuration="23.139348607s" totalImagesPullingTime="20.590158934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.13891125 +0000 UTC m=+23.869586342" watchObservedRunningTime="2026-07-23 18:16:37.139348607 +0000 UTC m=+23.870023697" Jul 23 18:16:37.156777 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.156745 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7ppq8" podStartSLOduration=2.445359142 podStartE2EDuration="23.156734301s" totalImagesPullingTime="20.711375159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.156609274 +0000 UTC m=+23.887284392" watchObservedRunningTime="2026-07-23 18:16:37.156734301 +0000 UTC m=+23.887409391" Jul 23 18:16:37.172091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.172043 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6bnvn" podStartSLOduration=2.285116988 podStartE2EDuration="23.172024586s" totalImagesPullingTime="20.886907598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.171377582 +0000 UTC m=+23.902052687" watchObservedRunningTime="2026-07-23 18:16:37.172024586 +0000 UTC m=+23.902699677" Jul 23 18:16:37.187823 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.187780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zw522" podStartSLOduration=2.329567306 podStartE2EDuration="23.187768551s" totalImagesPullingTime="20.858201245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:37.187551454 +0000 UTC m=+23.918226576" watchObservedRunningTime="2026-07-23 18:16:37.187768551 +0000 UTC m=+23.918443665" Jul 23 18:16:37.723147 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.723117 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:37.723754 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:37.723730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:38.004406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.004354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:38.077099 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.077071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p749r" event={"ID":"3b022432-5ff4-48d8-8935-a5d79acbccbd","Type":"ContainerStarted","Data":"4800db6d20a0d3296830ee9c3aedaba2b84ebb7fb92ec37ccc675f0bdd9de37d"} Jul 23 18:16:38.078840 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.078810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" event={"ID":"04b86fa819b797170650f264b6143415","Type":"ContainerStarted","Data":"f9c69f0c4917059909b09255a9926497a3f31b8847923a80c196a0009f36e041"} Jul 23 18:16:38.079791 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.079772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:38.080067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.080027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7ppq8" Jul 23 18:16:38.085043 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.085015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:38.106505 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.106460 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p749r" podStartSLOduration=3.21926922 podStartE2EDuration="24.106448628s" totalImagesPullingTime="20.887179408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:38.092385695 +0000 UTC m=+24.823060785" watchObservedRunningTime="2026-07-23 18:16:38.106448628 +0000 UTC m=+24.837123719" Jul 23 18:16:38.106886 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.106865 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-127.ec2.internal" podStartSLOduration=23.106857656 podStartE2EDuration="23.106857656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:38.106577683 +0000 UTC m=+24.837252772" watchObservedRunningTime="2026-07-23 18:16:38.106857656 +0000 UTC m=+24.837532747" Jul 23 18:16:38.129943 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.129926 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 18:16:38.185679 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.185652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:38.243048 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.243021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:38.752311 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.752197 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T18:16:38.129939811Z","UUID":"959333f6-894a-41c3-9f0e-52817020ede8","Handler":null,"Name":"","Endpoint":""} Jul 23 18:16:38.753926 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.753905 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 18:16:38.754063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.753941 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 18:16:38.828437 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.828403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:38.828629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.828501 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:38.828629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:38.828529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:38.828629 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:38.828523 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:38.828629 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:38.828611 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:38.828769 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:38.828676 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:39.082840 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:39.082771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" event={"ID":"ca54f61e-43d9-4fef-b083-44ca41a44534","Type":"ContainerStarted","Data":"a1f6fbf42ba9a08cdbafc101ab01cce31324eb9f7c42bf31e5cb67379f3405f6"} Jul 23 18:16:40.091133 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.091105 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:16:40.091707 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.091533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"178421d6672fb31a88953bdc8bf9e751ed3eef7a302c532e92cc8c3e49a02f88"} Jul 23 18:16:40.093712 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.093675 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" event={"ID":"ca54f61e-43d9-4fef-b083-44ca41a44534","Type":"ContainerStarted","Data":"d28b5f7311a189805456cf8e2eaba007956afc071c0c29364b0f7d091594815e"} Jul 23 18:16:40.112152 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.112114 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qs7ch" podStartSLOduration=3.015703747 podStartE2EDuration="26.112102212s" totalImagesPullingTime="23.096398465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:40.111574315 +0000 UTC m=+26.842249426" watchObservedRunningTime="2026-07-23 18:16:40.112102212 +0000 UTC m=+26.842777302" Jul 23 18:16:40.828570 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.828541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:40.828738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.828541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:40.828738 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:40.828645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:40.828738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.828541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:40.828878 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:40.828740 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:40.828878 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:40.828824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:40.914904 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:40.914879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:41.011548 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:41.011518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:41.996143 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:41.995893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:42.049464 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.049440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:42.100960 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.100943 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:16:42.101327 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.101308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"66ac383e23e1114f541255af7f929ac16b99c278ffdbaeeb91b289f2a1b3188c"} Jul 23 18:16:42.101635 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.101613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:42.101742 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.101707 2576 scope.go:117] "RemoveContainer" containerID="e7e3c91f57e26aa1abd08653a17fca30ce6d17a836f606b1ced2389b2758a8c4" Jul 23 18:16:42.118546 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.118530 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:42.828813 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.828784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:42.828995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.828784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:42.828995 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:42.828920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:42.828995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:42.828967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:42.829156 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:42.829122 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:42.829285 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:42.829255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:43.108256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.108217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:16:43.108729 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.108694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" event={"ID":"17b9081b-b362-48a0-8fb0-c53eaf57438e","Type":"ContainerStarted","Data":"7b5de8030e7e20a151676df064ad078d16fca1ccf69bfdee415c28300e003e6c"} Jul 23 18:16:43.109098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.108960 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:43.109098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.109048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:43.125943 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.125921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:16:43.136982 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.136828 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" podStartSLOduration=7.894978658 podStartE2EDuration="29.136811548s" totalImagesPullingTime="21.24183289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:43.135770034 +0000 UTC m=+29.866445174" watchObservedRunningTime="2026-07-23 18:16:43.136811548 +0000 UTC m=+29.867486641" Jul 23 18:16:43.294885 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.294846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdlln"] Jul 23 18:16:43.295032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.294964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:43.295092 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:43.295071 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:43.295671 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.295646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7glb"] Jul 23 18:16:43.295799 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.295735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:43.295851 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:43.295827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:43.296986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.296958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wc7ll"] Jul 23 18:16:43.297072 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:43.297060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:43.297193 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:43.297159 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:44.828886 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:44.828839 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:44.829455 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:44.828896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:44.829455 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:44.828959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:44.829455 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:44.828989 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:44.829455 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:44.829055 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:44.829455 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:44.829131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:45.114635 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:45.114569 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="e8afcef9aa7cba2d5d4e38aa521eddf047bb50dfa1f2c36107a222389e20fbb8" exitCode=0 Jul 23 18:16:45.114834 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:45.114661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"e8afcef9aa7cba2d5d4e38aa521eddf047bb50dfa1f2c36107a222389e20fbb8"} Jul 23 18:16:45.235204 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:45.235176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:45.459990 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:45.459962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:45.515906 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:45.515878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:46.118796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.118546 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="612177ab3defcc4da3b919ccb6e70676029748b046edbb30dfb8e4895b5b9fa0" exitCode=0 Jul 23 18:16:46.118796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.118635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"612177ab3defcc4da3b919ccb6e70676029748b046edbb30dfb8e4895b5b9fa0"} Jul 23 18:16:46.207532 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.207501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:46.284407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.284379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:46.536207 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.536178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:46.536343 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.536218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:46.536343 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.536322 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:46.536422 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.536375 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret podName:b31242ed-e9d8-4a85-b061-2b226f509c93 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.536359069 +0000 UTC m=+65.267034138 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret") pod "global-pull-secret-syncer-wdlln" (UID: "b31242ed-e9d8-4a85-b061-2b226f509c93") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:16:46.536422 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.536322 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:46.536493 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.536457 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs podName:17732286-b41e-4c5d-913a-440a7677d70d nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.536444123 +0000 UTC m=+65.267119196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs") pod "network-metrics-daemon-wc7ll" (UID: "17732286-b41e-4c5d-913a-440a7677d70d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:16:46.636618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.636592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:46.636750 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.636735 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:16:46.636821 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.636753 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:16:46.636821 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.636762 2576 projected.go:196] Error preparing data for projected volume kube-api-access-xdzks for pod openshift-network-diagnostics/network-check-target-g7glb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:46.636821 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.636811 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks podName:38288d42-be78-49db-908d-a5d10bbca786 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:18.636796525 +0000 UTC m=+65.367471593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdzks" (UniqueName: "kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks") pod "network-check-target-g7glb" (UID: "38288d42-be78-49db-908d-a5d10bbca786") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:16:46.670942 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.670905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:46.828338 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.828315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:46.828427 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.828315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:46.828461 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.828421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:46.828539 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.828515 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:46.828582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:46.828316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:46.828640 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:46.828623 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:47.123778 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:47.123710 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="70ec44e791175c4cc19a6d1e51e41bcc66f341c7cf81cdb13f8f0683036d7034" exitCode=0 Jul 23 18:16:47.124079 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:47.123781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"70ec44e791175c4cc19a6d1e51e41bcc66f341c7cf81cdb13f8f0683036d7034"} Jul 23 18:16:47.695212 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:47.695180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:48.829187 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:48.829157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:48.829674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:48.829158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:48.829674 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:48.829295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdlln" podUID="b31242ed-e9d8-4a85-b061-2b226f509c93" Jul 23 18:16:48.829674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:48.829164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:48.829674 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:48.829355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wc7ll" podUID="17732286-b41e-4c5d-913a-440a7677d70d" Jul 23 18:16:48.829674 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:48.829427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7glb" podUID="38288d42-be78-49db-908d-a5d10bbca786" Jul 23 18:16:49.130833 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.130762 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-127.ec2.internal" event="NodeReady" Jul 23 18:16:49.130988 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.130892 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 18:16:49.184289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.184206 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rhlsn"] Jul 23 18:16:49.189453 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.189429 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hpt6w"] Jul 23 18:16:49.189566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.189518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.192884 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.192861 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 18:16:49.193129 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.193112 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4ck5\"" Jul 23 18:16:49.193325 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.193112 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 18:16:49.194129 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.193785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.195794 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.195775 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 18:16:49.196256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.196220 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 18:16:49.196378 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.196286 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zbtqj\"" Jul 23 18:16:49.196378 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.196361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 18:16:49.201175 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.201136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rhlsn"] Jul 23 18:16:49.204046 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.204026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpt6w"] Jul 23 18:16:49.348809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.348773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:49.360811 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc9a111-88cb-4371-a4b7-2a8161730ec4-config-volume\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.360953 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/104c6971-882c-4298-9de3-7a9f2bf3cc26-cert\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.360953 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.360953 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmvt\" (UniqueName: \"kubernetes.io/projected/4bc9a111-88cb-4371-a4b7-2a8161730ec4-kube-api-access-tsmvt\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.361098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc9a111-88cb-4371-a4b7-2a8161730ec4-tmp-dir\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.361098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.360981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddlg\" (UniqueName: \"kubernetes.io/projected/104c6971-882c-4298-9de3-7a9f2bf3cc26-kube-api-access-6ddlg\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.410839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.410772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:49.462053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmvt\" (UniqueName: \"kubernetes.io/projected/4bc9a111-88cb-4371-a4b7-2a8161730ec4-kube-api-access-tsmvt\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.462191 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc9a111-88cb-4371-a4b7-2a8161730ec4-tmp-dir\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.462191 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddlg\" (UniqueName: \"kubernetes.io/projected/104c6971-882c-4298-9de3-7a9f2bf3cc26-kube-api-access-6ddlg\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc9a111-88cb-4371-a4b7-2a8161730ec4-config-volume\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/104c6971-882c-4298-9de3-7a9f2bf3cc26-cert\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:49.462419 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:49.462580 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:49.962456495 +0000 UTC m=+36.693131565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:16:49.462809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.462639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc9a111-88cb-4371-a4b7-2a8161730ec4-tmp-dir\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.464987 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.463367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc9a111-88cb-4371-a4b7-2a8161730ec4-config-volume\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.467493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.467471 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/104c6971-882c-4298-9de3-7a9f2bf3cc26-cert\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.471370 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.471350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddlg\" (UniqueName: \"kubernetes.io/projected/104c6971-882c-4298-9de3-7a9f2bf3cc26-kube-api-access-6ddlg\") pod \"ingress-canary-hpt6w\" (UID: \"104c6971-882c-4298-9de3-7a9f2bf3cc26\") " pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.471459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.471392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmvt\" (UniqueName: \"kubernetes.io/projected/4bc9a111-88cb-4371-a4b7-2a8161730ec4-kube-api-access-tsmvt\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.509702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.509683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpt6w" Jul 23 18:16:49.663898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.663733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpt6w"] Jul 23 18:16:49.964589 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:49.964550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:49.965409 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:49.964684 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:16:49.965409 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:49.964759 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:50.964738199 +0000 UTC m=+37.695413282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:16:50.334847 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.334741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:50.464487 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.464440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:50.829306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.829268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:16:50.829306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.829292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:16:50.829510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.829382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:16:50.832112 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.832087 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 18:16:50.832247 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.832092 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsfk8\"" Jul 23 18:16:50.832247 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.832137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 18:16:50.832945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.832925 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 18:16:50.833059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.833026 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7wd2\"" Jul 23 18:16:50.833112 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.833067 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 18:16:50.971980 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:50.971953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:50.972433 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:50.972103 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:16:50.972433 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:50.972169 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:52.972151743 +0000 UTC m=+39.702826815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:16:51.142045 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:51.141972 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104c6971_882c_4298_9de3_7a9f2bf3cc26.slice/crio-f284ff118a2b1d7cdcc2a85644bb1257633cbdd74bfcbdf3039aa719c7af8220 WatchSource:0}: Error finding container f284ff118a2b1d7cdcc2a85644bb1257633cbdd74bfcbdf3039aa719c7af8220: Status 404 returned error can't find the container with id f284ff118a2b1d7cdcc2a85644bb1257633cbdd74bfcbdf3039aa719c7af8220 Jul 23 18:16:52.137345 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.137314 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="78c6efb47dfaac129bc0e7e838810d7a4548d0fb5e859f35cdb54182660c0663" exitCode=0 Jul 23 18:16:52.137888 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.137392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"78c6efb47dfaac129bc0e7e838810d7a4548d0fb5e859f35cdb54182660c0663"} Jul 23 18:16:52.138556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.138531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpt6w" event={"ID":"104c6971-882c-4298-9de3-7a9f2bf3cc26","Type":"ContainerStarted","Data":"f284ff118a2b1d7cdcc2a85644bb1257633cbdd74bfcbdf3039aa719c7af8220"} Jul 23 18:16:52.604005 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.603971 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84b4c5997d-tvj7q"] Jul 23 18:16:52.614457 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.614436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.616881 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.616853 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 18:16:52.617021 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.616859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 18:16:52.618168 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.617668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 18:16:52.621396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.619167 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 18:16:52.621396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.619644 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 18:16:52.621396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.619896 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dr9c8\"" Jul 23 18:16:52.621396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.619914 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 18:16:52.622713 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.622692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84b4c5997d-tvj7q"] Jul 23 18:16:52.703810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.703779 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz"] Jul 23 18:16:52.717334 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.717312 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz"] Jul 23 18:16:52.717439 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.717406 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" Jul 23 18:16:52.719791 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.719774 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vqwtn\"" Jul 23 18:16:52.741584 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.741564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:52.785718 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.785695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.785796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.785719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-stats-auth\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.785796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.785737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw5k\" (UniqueName: \"kubernetes.io/projected/f9200722-028b-4a55-a47f-94b718cbedaa-kube-api-access-vmw5k\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.785796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.785761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.785900 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.785832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-default-certificate\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.806580 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.806559 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt"] Jul 23 18:16:52.810995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.810968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:52.813133 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.813107 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg"] Jul 23 18:16:52.813238 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.813214 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" Jul 23 18:16:52.816433 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.816412 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:16:52.816541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.816456 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 18:16:52.816642 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.816618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hqqcr\"" Jul 23 18:16:52.819337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.819299 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ktvjm"] Jul 23 18:16:52.820077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.819910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:52.822939 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.822733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 18:16:52.824515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.824261 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 18:16:52.824515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.824333 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f6ntr\"" Jul 23 18:16:52.829463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.829394 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt"] Jul 23 18:16:52.829463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.829426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ktvjm"] Jul 23 18:16:52.829463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.829446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg"] Jul 23 18:16:52.829668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.829525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:52.831945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.831925 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:16:52.832176 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.832157 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l74wl\"" Jul 23 18:16:52.832304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.832285 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 18:16:52.832420 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.832389 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 18:16:52.832522 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.832513 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 18:16:52.838988 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.838970 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 18:16:52.886365 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-default-certificate\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.886476 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbww7\" (UniqueName: \"kubernetes.io/projected/b0400051-4c59-4087-a849-9fe24acb44f0-kube-api-access-gbww7\") pod \"network-check-source-7bc5b6bf85-qbgjz\" (UID: \"b0400051-4c59-4087-a849-9fe24acb44f0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" Jul 23 18:16:52.886476 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.886476 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-stats-auth\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.886634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw5k\" (UniqueName: \"kubernetes.io/projected/f9200722-028b-4a55-a47f-94b718cbedaa-kube-api-access-vmw5k\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.886634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.886532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.886634 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:52.886624 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 18:16:52.886784 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:52.886676 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:53.386658554 +0000 UTC m=+40.117333624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : secret "router-metrics-certs-default" not found Jul 23 18:16:52.887280 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:52.887064 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:53.387036054 +0000 UTC m=+40.117711130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : configmap references non-existent config key: service-ca.crt Jul 23 18:16:52.890354 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.890311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-stats-auth\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.890452 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.890392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-default-certificate\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.897562 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.897520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw5k\" (UniqueName: \"kubernetes.io/projected/f9200722-028b-4a55-a47f-94b718cbedaa-kube-api-access-vmw5k\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:52.908410 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.908384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:16:52.920313 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.920294 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:52.922801 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.922776 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 18:16:52.923144 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.923130 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 18:16:52.923218 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.923181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ptt8h\"" Jul 23 18:16:52.923529 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.923508 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 18:16:52.928475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.928441 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:16:52.932155 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.931881 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 18:16:52.948982 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.948953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:52.987780 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:52.987868 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-trusted-ca\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:52.987868 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbq66\" (UniqueName: \"kubernetes.io/projected/b2904bb2-0993-4eb8-8d07-2194054515f6-kube-api-access-cbq66\") pod \"volume-data-source-validator-78cf4c6f6-kqvbt\" (UID: \"b2904bb2-0993-4eb8-8d07-2194054515f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" Jul 23 18:16:52.987957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5md\" (UniqueName: \"kubernetes.io/projected/ab6594f7-2fae-4ad8-b3b0-7a08931be111-kube-api-access-nf5md\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:52.987957 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:52.987891 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:16:52.987957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45587963-6805-4d17-b098-64a886a354cc-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:52.987957 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:52.987947 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:56.987928869 +0000 UTC m=+43.718603938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:16:52.988125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.987986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6594f7-2fae-4ad8-b3b0-7a08931be111-serving-cert\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:52.988125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.988018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:52.988125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.988070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbww7\" (UniqueName: \"kubernetes.io/projected/b0400051-4c59-4087-a849-9fe24acb44f0-kube-api-access-gbww7\") pod \"network-check-source-7bc5b6bf85-qbgjz\" (UID: \"b0400051-4c59-4087-a849-9fe24acb44f0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" Jul 23 18:16:52.988125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.988094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-config\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:52.997040 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:52.997011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbww7\" (UniqueName: \"kubernetes.io/projected/b0400051-4c59-4087-a849-9fe24acb44f0-kube-api-access-gbww7\") pod \"network-check-source-7bc5b6bf85-qbgjz\" (UID: \"b0400051-4c59-4087-a849-9fe24acb44f0\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" Jul 23 18:16:53.025924 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.025902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" Jul 23 18:16:53.045472 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.045443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:53.088418 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.088526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5md\" (UniqueName: \"kubernetes.io/projected/ab6594f7-2fae-4ad8-b3b0-7a08931be111-kube-api-access-nf5md\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.088526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.088605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6594f7-2fae-4ad8-b3b0-7a08931be111-serving-cert\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.088605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq66\" (UniqueName: \"kubernetes.io/projected/b2904bb2-0993-4eb8-8d07-2194054515f6-kube-api-access-cbq66\") pod \"volume-data-source-validator-78cf4c6f6-kqvbt\" (UID: \"b2904bb2-0993-4eb8-8d07-2194054515f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" Jul 23 18:16:53.088605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52mh\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.088693 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.088629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-trusted-ca\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.089337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.089337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.089337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.089560 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.089560 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45587963-6805-4d17-b098-64a886a354cc-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:53.089560 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:53.089560 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-config\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.089560 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.089800 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.089641 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-trusted-ca\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.089800 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.089727 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:16:53.089800 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.089779 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:16:53.589764191 +0000 UTC m=+40.320439261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:16:53.090415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.090364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/45587963-6805-4d17-b098-64a886a354cc-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:53.090415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.090364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ab6594f7-2fae-4ad8-b3b0-7a08931be111-config\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.092824 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.092603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ab6594f7-2fae-4ad8-b3b0-7a08931be111-serving-cert\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.097242 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.097211 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq66\" (UniqueName: \"kubernetes.io/projected/b2904bb2-0993-4eb8-8d07-2194054515f6-kube-api-access-cbq66\") pod \"volume-data-source-validator-78cf4c6f6-kqvbt\" (UID: \"b2904bb2-0993-4eb8-8d07-2194054515f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" Jul 23 18:16:53.098171 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.098151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5md\" (UniqueName: \"kubernetes.io/projected/ab6594f7-2fae-4ad8-b3b0-7a08931be111-kube-api-access-nf5md\") pod \"console-operator-565b7b569d-ktvjm\" (UID: \"ab6594f7-2fae-4ad8-b3b0-7a08931be111\") " pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.127854 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.127827 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" Jul 23 18:16:53.143731 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.143707 2576 generic.go:356] "Generic (PLEG): container finished" podID="a4ec536d-8503-4045-bae0-908bd6c9f99a" containerID="2fc4ca3ac68564f59c54e075652d8e95f795ff50282e1bba564495ca9932656d" exitCode=0 Jul 23 18:16:53.144023 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.143744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerDied","Data":"2fc4ca3ac68564f59c54e075652d8e95f795ff50282e1bba564495ca9932656d"} Jul 23 18:16:53.146064 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.146045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:16:53.170497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.170356 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz"] Jul 23 18:16:53.175822 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:53.175797 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0400051_4c59_4087_a849_9fe24acb44f0.slice/crio-65520776b268b1f61a7f22d87a2b74901b5fd106168264bfb45f43536f1a9bcd WatchSource:0}: Error finding container 65520776b268b1f61a7f22d87a2b74901b5fd106168264bfb45f43536f1a9bcd: Status 404 returned error can't find the container with id 65520776b268b1f61a7f22d87a2b74901b5fd106168264bfb45f43536f1a9bcd Jul 23 18:16:53.190520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52mh\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.190977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.191263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.191669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.191564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.192619 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.192330 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:16:53.192619 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.192350 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbf98c45-xhghl: secret "image-registry-tls" not found Jul 23 18:16:53.192619 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.192404 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls podName:eee0b41e-0f41-473c-99e4-79e955613805 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:53.692384832 +0000 UTC m=+40.423059921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls") pod "image-registry-75fbf98c45-xhghl" (UID: "eee0b41e-0f41-473c-99e4-79e955613805") : secret "image-registry-tls" not found Jul 23 18:16:53.193307 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.193289 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.198032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.195747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.199306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.199113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.200618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.200575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.201103 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.201084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52mh\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.216462 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.216363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:53.278035 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.277727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:53.327362 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.327336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt"] Jul 23 18:16:53.330150 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:53.330120 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2904bb2_0993_4eb8_8d07_2194054515f6.slice/crio-660247a680686b8addfe03b0f3bcbc5e49f30b5ec674fe0aceffa101d8ae74d0 WatchSource:0}: Error finding container 660247a680686b8addfe03b0f3bcbc5e49f30b5ec674fe0aceffa101d8ae74d0: Status 404 returned error can't find the container with id 660247a680686b8addfe03b0f3bcbc5e49f30b5ec674fe0aceffa101d8ae74d0 Jul 23 18:16:53.350415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.350392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ktvjm"] Jul 23 18:16:53.355587 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:53.355560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6594f7_2fae_4ad8_b3b0_7a08931be111.slice/crio-dcb537aae206b40da205830aaf8eef34a5ca5060539adae8fa239be34201457c WatchSource:0}: Error finding container dcb537aae206b40da205830aaf8eef34a5ca5060539adae8fa239be34201457c: Status 404 returned error can't find the container with id dcb537aae206b40da205830aaf8eef34a5ca5060539adae8fa239be34201457c Jul 23 18:16:53.392869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.392811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:53.392869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.392841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:53.393022 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.393009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.392972175 +0000 UTC m=+41.123647247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : configmap references non-existent config key: service-ca.crt Jul 23 18:16:53.393068 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.393036 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 18:16:53.393105 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.393085 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.393072325 +0000 UTC m=+41.123747395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : secret "router-metrics-certs-default" not found Jul 23 18:16:53.594841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.594799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:53.595016 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.594964 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:16:53.595059 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.595031 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.595016939 +0000 UTC m=+41.325692011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:16:53.695204 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:53.695174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:53.695370 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.695329 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:16:53.695370 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.695352 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbf98c45-xhghl: secret "image-registry-tls" not found Jul 23 18:16:53.695481 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:53.695408 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls podName:eee0b41e-0f41-473c-99e4-79e955613805 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:54.695388435 +0000 UTC m=+41.426063508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls") pod "image-registry-75fbf98c45-xhghl" (UID: "eee0b41e-0f41-473c-99e4-79e955613805") : secret "image-registry-tls" not found Jul 23 18:16:54.148430 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.148339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" event={"ID":"ab6594f7-2fae-4ad8-b3b0-7a08931be111","Type":"ContainerStarted","Data":"dcb537aae206b40da205830aaf8eef34a5ca5060539adae8fa239be34201457c"} Jul 23 18:16:54.149876 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.149849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" event={"ID":"b2904bb2-0993-4eb8-8d07-2194054515f6","Type":"ContainerStarted","Data":"660247a680686b8addfe03b0f3bcbc5e49f30b5ec674fe0aceffa101d8ae74d0"} Jul 23 18:16:54.153543 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.153507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrf52" event={"ID":"a4ec536d-8503-4045-bae0-908bd6c9f99a","Type":"ContainerStarted","Data":"22989c1fb731fb2db3568fadf90406f4c8bdfded84e30d63f35bc1797780dfcf"} Jul 23 18:16:54.155006 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.154947 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" event={"ID":"b0400051-4c59-4087-a849-9fe24acb44f0","Type":"ContainerStarted","Data":"65520776b268b1f61a7f22d87a2b74901b5fd106168264bfb45f43536f1a9bcd"} Jul 23 18:16:54.296004 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.295809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:54.396153 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.396113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:54.402115 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.402051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:54.402115 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.402096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:54.402315 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.402203 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:56.402181938 +0000 UTC m=+43.132857009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : configmap references non-existent config key: service-ca.crt Jul 23 18:16:54.402315 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.402289 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 18:16:54.402429 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.402333 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:16:56.402321487 +0000 UTC m=+43.132996556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : secret "router-metrics-certs-default" not found Jul 23 18:16:54.496383 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.496352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:54.596956 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.596924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:54.604026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.603991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:54.604181 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.604157 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:16:54.604288 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.604251 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:16:56.604215174 +0000 UTC m=+43.334890260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:16:54.705445 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:54.705412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:54.705575 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.705540 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:16:54.705575 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.705559 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbf98c45-xhghl: secret "image-registry-tls" not found Jul 23 18:16:54.705687 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:54.705616 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls podName:eee0b41e-0f41-473c-99e4-79e955613805 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:56.705600104 +0000 UTC m=+43.436275192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls") pod "image-registry-75fbf98c45-xhghl" (UID: "eee0b41e-0f41-473c-99e4-79e955613805") : secret "image-registry-tls" not found Jul 23 18:16:56.161718 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.161654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpt6w" event={"ID":"104c6971-882c-4298-9de3-7a9f2bf3cc26","Type":"ContainerStarted","Data":"bcb89b5d435a7b0ed38e2edbdda9ee516d6a8629768b4c141851e6350768effe"} Jul 23 18:16:56.179077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.179021 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hpt6w" podStartSLOduration=2.8799093669999998 podStartE2EDuration="7.179000188s" totalImagesPullingTime="4.299090821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:56.178312682 +0000 UTC m=+42.908987775" watchObservedRunningTime="2026-07-23 18:16:56.179000188 +0000 UTC m=+42.909675281" Jul 23 18:16:56.179746 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.179639 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vrf52" podStartSLOduration=8.767553181 podStartE2EDuration="42.179626475s" totalImagesPullingTime="33.412073294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:54.176453293 +0000 UTC m=+40.907128429" watchObservedRunningTime="2026-07-23 18:16:56.179626475 +0000 UTC m=+42.910301569" Jul 23 18:16:56.419272 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.419178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:56.419272 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.419242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:16:56.419491 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.419365 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 18:16:56.419491 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.419421 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.419405956 +0000 UTC m=+47.150081026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : secret "router-metrics-certs-default" not found Jul 23 18:16:56.419749 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.419719 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.419700137 +0000 UTC m=+47.150375221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : configmap references non-existent config key: service-ca.crt Jul 23 18:16:56.620983 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.620939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:16:56.621142 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.621115 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:16:56.621207 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.621195 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.621174343 +0000 UTC m=+47.351849426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:16:56.721613 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:56.721582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:16:56.721750 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.721708 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:16:56.721750 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.721726 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbf98c45-xhghl: secret "image-registry-tls" not found Jul 23 18:16:56.721844 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:56.721777 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls podName:eee0b41e-0f41-473c-99e4-79e955613805 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.721760677 +0000 UTC m=+47.452435746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls") pod "image-registry-75fbf98c45-xhghl" (UID: "eee0b41e-0f41-473c-99e4-79e955613805") : secret "image-registry-tls" not found Jul 23 18:16:57.025146 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:57.025121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:16:57.025275 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:57.025262 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:16:57.025330 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:57.025319 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:05.025304886 +0000 UTC m=+51.755979958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:16:57.348928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:57.348848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.039966 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.039926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.153067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.153040 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.168174 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.168156 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/0.log" Jul 23 18:16:58.168292 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.168190 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab6594f7-2fae-4ad8-b3b0-7a08931be111" containerID="77685fc55b15dfeaa7f0f2b42597a7ac8e22fb459f5d8aaa5fb62ca62b55a230" exitCode=255 Jul 23 18:16:58.168292 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.168255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" event={"ID":"ab6594f7-2fae-4ad8-b3b0-7a08931be111","Type":"ContainerDied","Data":"77685fc55b15dfeaa7f0f2b42597a7ac8e22fb459f5d8aaa5fb62ca62b55a230"} Jul 23 18:16:58.168509 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.168492 2576 scope.go:117] "RemoveContainer" containerID="77685fc55b15dfeaa7f0f2b42597a7ac8e22fb459f5d8aaa5fb62ca62b55a230" Jul 23 18:16:58.169567 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.169550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" event={"ID":"b2904bb2-0993-4eb8-8d07-2194054515f6","Type":"ContainerStarted","Data":"fa2aca909c0819c2514e141a791b6b6a460cb86f61efa322e80140207458c0cd"} Jul 23 18:16:58.222067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.221985 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-kqvbt" podStartSLOduration=1.5971942430000001 podStartE2EDuration="6.221974139s" totalImagesPullingTime="4.624779896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:58.221813184 +0000 UTC m=+44.952488286" watchObservedRunningTime="2026-07-23 18:16:58.221974139 +0000 UTC m=+44.952649229" Jul 23 18:16:58.245935 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.245909 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4"] Jul 23 18:16:58.259936 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.259915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" Jul 23 18:16:58.260635 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.260613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4"] Jul 23 18:16:58.262612 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.262558 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 18:16:58.262612 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.262583 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lj6l5\"" Jul 23 18:16:58.262783 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.262564 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 18:16:58.267680 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.267400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.359379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.359361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.437831 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.437812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp55h\" (UniqueName: \"kubernetes.io/projected/03025fe2-d410-4448-b76a-b271b2335c8f-kube-api-access-gp55h\") pod \"migrator-7cc9c7ffd7-phlf4\" (UID: \"03025fe2-d410-4448-b76a-b271b2335c8f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" Jul 23 18:16:58.456566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.456539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.539191 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.539167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp55h\" (UniqueName: \"kubernetes.io/projected/03025fe2-d410-4448-b76a-b271b2335c8f-kube-api-access-gp55h\") pod \"migrator-7cc9c7ffd7-phlf4\" (UID: \"03025fe2-d410-4448-b76a-b271b2335c8f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" Jul 23 18:16:58.549347 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.549326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp55h\" (UniqueName: \"kubernetes.io/projected/03025fe2-d410-4448-b76a-b271b2335c8f-kube-api-access-gp55h\") pod \"migrator-7cc9c7ffd7-phlf4\" (UID: \"03025fe2-d410-4448-b76a-b271b2335c8f\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" Jul 23 18:16:58.556606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.556584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.570445 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.570419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" Jul 23 18:16:58.715291 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.715257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.724531 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.724504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4"] Jul 23 18:16:58.728646 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:16:58.728623 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03025fe2_d410_4448_b76a_b271b2335c8f.slice/crio-7b907fdd21341d7e791363a34d2bba9f7272c4dcee9bef9797fb2714b37bdc08 WatchSource:0}: Error finding container 7b907fdd21341d7e791363a34d2bba9f7272c4dcee9bef9797fb2714b37bdc08: Status 404 returned error can't find the container with id 7b907fdd21341d7e791363a34d2bba9f7272c4dcee9bef9797fb2714b37bdc08 Jul 23 18:16:58.819999 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.819938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:58.912960 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.912937 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n6f"] Jul 23 18:16:58.942172 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.942151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n6f"] Jul 23 18:16:58.942298 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.942285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:58.945887 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.945870 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 18:16:58.945991 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.945888 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 18:16:58.945991 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.945889 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 18:16:58.946217 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.946199 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 18:16:58.946361 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.946347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ccngx\"" Jul 23 18:16:58.967120 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:58.967094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:59.042776 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.042752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bd220f8-70f0-491d-b362-cb4d9db633f4-crio-socket\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.042883 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.042866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.042944 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.042914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bd220f8-70f0-491d-b362-cb4d9db633f4-data-volume\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.042981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.042965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zlwv\" (UniqueName: \"kubernetes.io/projected/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-api-access-5zlwv\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.043063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.043045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.065140 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.065121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:16:59.143344 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143344 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bd220f8-70f0-491d-b362-cb4d9db633f4-data-volume\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143480 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zlwv\" (UniqueName: \"kubernetes.io/projected/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-api-access-5zlwv\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143480 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.143416 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:16:59.143480 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143480 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.143467 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:59.643449258 +0000 UTC m=+46.374124327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : configmap "kube-rbac-proxy" not found Jul 23 18:16:59.143689 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bd220f8-70f0-491d-b362-cb4d9db633f4-crio-socket\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143689 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.143518 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:16:59.143689 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.143555 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:16:59.643544983 +0000 UTC m=+46.374220059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : secret "insights-runtime-extractor-tls" not found Jul 23 18:16:59.143689 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bd220f8-70f0-491d-b362-cb4d9db633f4-data-volume\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.143841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.143659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1bd220f8-70f0-491d-b362-cb4d9db633f4-crio-socket\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.151921 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.151896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zlwv\" (UniqueName: \"kubernetes.io/projected/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-api-access-5zlwv\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.173716 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.173692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" event={"ID":"b0400051-4c59-4087-a849-9fe24acb44f0","Type":"ContainerStarted","Data":"4862d96b926f8e33c31f5eb1cdb950b7d37d31c8df97c716a912b27ebbb26d69"} Jul 23 18:16:59.174813 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.174792 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" event={"ID":"03025fe2-d410-4448-b76a-b271b2335c8f","Type":"ContainerStarted","Data":"7b907fdd21341d7e791363a34d2bba9f7272c4dcee9bef9797fb2714b37bdc08"} Jul 23 18:16:59.176198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176182 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:16:59.176608 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/0.log" Jul 23 18:16:59.176693 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176628 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab6594f7-2fae-4ad8-b3b0-7a08931be111" containerID="338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912" exitCode=255 Jul 23 18:16:59.176753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" event={"ID":"ab6594f7-2fae-4ad8-b3b0-7a08931be111","Type":"ContainerDied","Data":"338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912"} Jul 23 18:16:59.176753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176739 2576 scope.go:117] "RemoveContainer" containerID="77685fc55b15dfeaa7f0f2b42597a7ac8e22fb459f5d8aaa5fb62ca62b55a230" Jul 23 18:16:59.176953 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.176927 2576 scope.go:117] "RemoveContainer" containerID="338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912" Jul 23 18:16:59.177114 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.177095 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ktvjm_openshift-console-operator(ab6594f7-2fae-4ad8-b3b0-7a08931be111)\"" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" podUID="ab6594f7-2fae-4ad8-b3b0-7a08931be111" Jul 23 18:16:59.190703 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.190663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-qbgjz" podStartSLOduration=1.988283641 podStartE2EDuration="7.190650668s" totalImagesPullingTime="5.202367027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:59.18986976 +0000 UTC m=+45.920544851" watchObservedRunningTime="2026-07-23 18:16:59.190650668 +0000 UTC m=+45.921325759" Jul 23 18:16:59.647846 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.647817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.648139 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:16:59.647896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:16:59.648139 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.647939 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:16:59.648139 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.647990 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:16:59.648139 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.648000 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.647986556 +0000 UTC m=+47.378661628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : secret "insights-runtime-extractor-tls" not found Jul 23 18:16:59.648139 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:16:59.648028 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:00.648017522 +0000 UTC m=+47.378692594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : configmap "kube-rbac-proxy" not found Jul 23 18:17:00.180966 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.180941 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:17:00.181321 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.181302 2576 scope.go:117] "RemoveContainer" containerID="338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912" Jul 23 18:17:00.181492 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.181471 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ktvjm_openshift-console-operator(ab6594f7-2fae-4ad8-b3b0-7a08931be111)\"" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" podUID="ab6594f7-2fae-4ad8-b3b0-7a08931be111" Jul 23 18:17:00.376909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.376877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:00.455489 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.455458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:00.455628 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.455493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:00.455628 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.455612 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 18:17:00.455698 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.455613 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:17:08.455590332 +0000 UTC m=+55.186265402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : configmap references non-existent config key: service-ca.crt Jul 23 18:17:00.455698 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.455667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs podName:f9200722-028b-4a55-a47f-94b718cbedaa nodeName:}" failed. No retries permitted until 2026-07-23 18:17:08.455654785 +0000 UTC m=+55.186329855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs") pod "router-default-84b4c5997d-tvj7q" (UID: "f9200722-028b-4a55-a47f-94b718cbedaa") : secret "router-metrics-certs-default" not found Jul 23 18:17:00.657205 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.657172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.657244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657306 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657305 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657365 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:17:08.65734931 +0000 UTC m=+55.388024399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657393 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:02.657383835 +0000 UTC m=+49.388058905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : configmap "kube-rbac-proxy" not found Jul 23 18:17:00.657523 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.657444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:00.657731 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657554 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:17:00.657731 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.657599 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:02.657586637 +0000 UTC m=+49.388261707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : secret "insights-runtime-extractor-tls" not found Jul 23 18:17:00.758809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.758693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:00.758916 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.758825 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 18:17:00.758916 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.758843 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75fbf98c45-xhghl: secret "image-registry-tls" not found Jul 23 18:17:00.758916 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:00.758904 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls podName:eee0b41e-0f41-473c-99e4-79e955613805 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:08.758891969 +0000 UTC m=+55.489567042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls") pod "image-registry-75fbf98c45-xhghl" (UID: "eee0b41e-0f41-473c-99e4-79e955613805") : secret "image-registry-tls" not found Jul 23 18:17:00.985360 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:00.985335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:01.072462 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:01.072434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bnvn_f169b268-cb81-47b7-a6ae-b4305a26c317/dns-node-resolver/0.log" Jul 23 18:17:01.187030 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:01.187002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" event={"ID":"03025fe2-d410-4448-b76a-b271b2335c8f","Type":"ContainerStarted","Data":"ec66a6f66de9cd0ef0b2d03614e540cda6ff5f4feb6dabe3ce35273fcf486a11"} Jul 23 18:17:01.187123 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:01.187056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" event={"ID":"03025fe2-d410-4448-b76a-b271b2335c8f","Type":"ContainerStarted","Data":"b5982c032f7256709617f63c72f2ce4fce2d19830e73b2d58749fab1016168c6"} Jul 23 18:17:01.211140 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:01.211103 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-phlf4" podStartSLOduration=1.133264405 podStartE2EDuration="3.211085754s" totalImagesPullingTime="2.077821349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:01.21098017 +0000 UTC m=+47.941655290" watchObservedRunningTime="2026-07-23 18:17:01.211085754 +0000 UTC m=+47.941760846" Jul 23 18:17:01.425406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:01.425351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.004450 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.004415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.072100 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.072077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcl42_6201bed0-3725-46e8-aa9a-25259f731d44/node-ca/0.log" Jul 23 18:17:02.198382 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.198335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.298177 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.298120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.398605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.398583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.469636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.469614 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:02.674189 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.674126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:02.674322 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.674199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:02.674322 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:02.674291 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:17:02.674322 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:02.674301 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:17:02.674450 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:02.674337 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:06.674324691 +0000 UTC m=+53.404999765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : configmap "kube-rbac-proxy" not found Jul 23 18:17:02.674450 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:02.674359 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:06.674342601 +0000 UTC m=+53.405017669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : secret "insights-runtime-extractor-tls" not found Jul 23 18:17:02.872097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:02.872067 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpt6w_104c6971-882c-4298-9de3-7a9f2bf3cc26/serve-healthcheck-canary/0.log" Jul 23 18:17:03.125750 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.125724 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:03.146944 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.146923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:17:03.147022 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.146993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:17:03.147294 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.147282 2576 scope.go:117] "RemoveContainer" containerID="338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912" Jul 23 18:17:03.147426 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:03.147410 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ktvjm_openshift-console-operator(ab6594f7-2fae-4ad8-b3b0-7a08931be111)\"" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" podUID="ab6594f7-2fae-4ad8-b3b0-7a08931be111" Jul 23 18:17:03.271874 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.271847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-phlf4_03025fe2-d410-4448-b76a-b271b2335c8f/migrator/0.log" Jul 23 18:17:03.315947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.315913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:03.472415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.472394 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-phlf4_03025fe2-d410-4448-b76a-b271b2335c8f/graceful-termination/0.log" Jul 23 18:17:03.900785 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:03.900694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:04.504412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:04.504383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:04.705546 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:04.705492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:05.093479 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.093452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:05.093616 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:05.093559 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:17:05.093616 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:05.093609 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls podName:4bc9a111-88cb-4371-a4b7-2a8161730ec4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:21.093595456 +0000 UTC m=+67.824270525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls") pod "dns-default-rhlsn" (UID: "4bc9a111-88cb-4371-a4b7-2a8161730ec4") : secret "dns-default-metrics-tls" not found Jul 23 18:17:05.100648 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.100622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:05.304768 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.304741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:05.719164 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.719128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:05.911779 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.911744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:05.977426 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:05.977361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:06.051528 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.051500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:06.303440 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.303373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:06.700871 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.700838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:06.705698 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.705675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:06.705795 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.705768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:06.705849 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:06.705834 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:17:06.705935 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:06.705924 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:14.705907791 +0000 UTC m=+61.436582865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : secret "insights-runtime-extractor-tls" not found Jul 23 18:17:06.705974 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:06.705834 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:17:06.706028 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:06.705979 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm podName:1bd220f8-70f0-491d-b362-cb4d9db633f4 nodeName:}" failed. No retries permitted until 2026-07-23 18:17:14.705965868 +0000 UTC m=+61.436640938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s4n6f" (UID: "1bd220f8-70f0-491d-b362-cb4d9db633f4") : configmap "kube-rbac-proxy" not found Jul 23 18:17:06.802112 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:06.802054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:07.102991 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:07.102913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:07.316688 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:07.316654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:07.705205 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:07.705166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:08.301948 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.301912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:08.410197 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.410168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:08.521514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.521484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:08.521514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.521515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:08.522215 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.522182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9200722-028b-4a55-a47f-94b718cbedaa-service-ca-bundle\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:08.524142 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.524115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9200722-028b-4a55-a47f-94b718cbedaa-metrics-certs\") pod \"router-default-84b4c5997d-tvj7q\" (UID: \"f9200722-028b-4a55-a47f-94b718cbedaa\") " pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:08.526967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.526942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:08.674556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.674530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84b4c5997d-tvj7q"] Jul 23 18:17:08.678158 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:08.678134 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9200722_028b_4a55_a47f_94b718cbedaa.slice/crio-dba5f0c0ca8eec9c8f966365d49bd0e90b9a2670f8c682859630f7d79532d521 WatchSource:0}: Error finding container dba5f0c0ca8eec9c8f966365d49bd0e90b9a2670f8c682859630f7d79532d521: Status 404 returned error can't find the container with id dba5f0c0ca8eec9c8f966365d49bd0e90b9a2670f8c682859630f7d79532d521 Jul 23 18:17:08.723081 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.723058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:17:08.723164 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:08.723156 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 18:17:08.723206 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:08.723200 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert podName:45587963-6805-4d17-b098-64a886a354cc nodeName:}" failed. No retries permitted until 2026-07-23 18:17:24.723186393 +0000 UTC m=+71.453861466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-n8gtg" (UID: "45587963-6805-4d17-b098-64a886a354cc") : secret "networking-console-plugin-cert" not found Jul 23 18:17:08.726787 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.726760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:08.823751 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.823666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:08.825895 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.825873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"image-registry-75fbf98c45-xhghl\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:08.829856 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.829840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:08.902175 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.902144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:08.957385 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:08.957350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:17:08.960756 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:08.960728 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee0b41e_0f41_473c_99e4_79e955613805.slice/crio-b9cd8e053a7763af140eea79a873ea6bbfd4c8908b680495c7f8959ecde8565f WatchSource:0}: Error finding container b9cd8e053a7763af140eea79a873ea6bbfd4c8908b680495c7f8959ecde8565f: Status 404 returned error can't find the container with id b9cd8e053a7763af140eea79a873ea6bbfd4c8908b680495c7f8959ecde8565f Jul 23 18:17:09.113848 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.113767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:09.211971 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.211939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" event={"ID":"f9200722-028b-4a55-a47f-94b718cbedaa","Type":"ContainerStarted","Data":"2aab26691d6a3414ceea6ec57d7a97c3f7ba56c04bce3640dfacc0f604723bdc"} Jul 23 18:17:09.212133 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.211995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" event={"ID":"f9200722-028b-4a55-a47f-94b718cbedaa","Type":"ContainerStarted","Data":"dba5f0c0ca8eec9c8f966365d49bd0e90b9a2670f8c682859630f7d79532d521"} Jul 23 18:17:09.213570 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.213546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" event={"ID":"eee0b41e-0f41-473c-99e4-79e955613805","Type":"ContainerStarted","Data":"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65"} Jul 23 18:17:09.213680 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.213586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" event={"ID":"eee0b41e-0f41-473c-99e4-79e955613805","Type":"ContainerStarted","Data":"b9cd8e053a7763af140eea79a873ea6bbfd4c8908b680495c7f8959ecde8565f"} Jul 23 18:17:09.213680 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.213667 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:09.230951 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.230914 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" podStartSLOduration=17.230901298 podStartE2EDuration="17.230901298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:09.229983974 +0000 UTC m=+55.960659068" watchObservedRunningTime="2026-07-23 18:17:09.230901298 +0000 UTC m=+55.961576388" Jul 23 18:17:09.247600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.247556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" podStartSLOduration=17.247540298 podStartE2EDuration="17.247540298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:09.246206611 +0000 UTC m=+55.976881714" watchObservedRunningTime="2026-07-23 18:17:09.247540298 +0000 UTC m=+55.978215389" Jul 23 18:17:09.508075 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.508007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:09.527908 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.527886 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:09.530643 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.530608 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:09.707104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:09.707074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:10.217305 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:10.217272 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:10.218571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:10.218543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84b4c5997d-tvj7q" Jul 23 18:17:11.152091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.152053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:11.252794 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.252762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:11.353368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.353336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:11.497571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.497537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:11.702932 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.702892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:11.902296 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:11.902197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:12.506267 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:12.506213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:12.924056 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:12.923975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:14.772370 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.772331 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:14.772753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.772396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:14.772841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.772819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1bd220f8-70f0-491d-b362-cb4d9db633f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:14.774719 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.774694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1bd220f8-70f0-491d-b362-cb4d9db633f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s4n6f\" (UID: \"1bd220f8-70f0-491d-b362-cb4d9db633f4\") " pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:14.829405 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.829384 2576 scope.go:117] "RemoveContainer" containerID="338eb44496f4f9289773b98942bd9f5944ee3f89ab100cf375ab29b81b47b912" Jul 23 18:17:14.853485 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.853463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ccngx\"" Jul 23 18:17:14.861889 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:14.861870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s4n6f" Jul 23 18:17:15.032433 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.032379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s4n6f"] Jul 23 18:17:15.035650 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:15.035617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd220f8_70f0_491d_b362_cb4d9db633f4.slice/crio-329c8f50a85de38c0548eec8c5bbc9317ab1bd073bc95ac8223d98028afae664 WatchSource:0}: Error finding container 329c8f50a85de38c0548eec8c5bbc9317ab1bd073bc95ac8223d98028afae664: Status 404 returned error can't find the container with id 329c8f50a85de38c0548eec8c5bbc9317ab1bd073bc95ac8223d98028afae664 Jul 23 18:17:15.129602 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.129578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcgwc" Jul 23 18:17:15.236957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.236936 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:17:15.237077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.237035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" event={"ID":"ab6594f7-2fae-4ad8-b3b0-7a08931be111","Type":"ContainerStarted","Data":"ea2a0087e48d6840e4d6725ee85ffa9c5aa9772361a58e3f3e4dcfccae12bdf8"} Jul 23 18:17:15.237309 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.237292 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:17:15.238248 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.238208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n6f" event={"ID":"1bd220f8-70f0-491d-b362-cb4d9db633f4","Type":"ContainerStarted","Data":"329c8f50a85de38c0548eec8c5bbc9317ab1bd073bc95ac8223d98028afae664"} Jul 23 18:17:15.252006 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.251970 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" podStartSLOduration=18.651653941 podStartE2EDuration="23.251958185s" totalImagesPullingTime="4.600304244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:15.250784182 +0000 UTC m=+61.981459303" watchObservedRunningTime="2026-07-23 18:17:15.251958185 +0000 UTC m=+61.982633273" Jul 23 18:17:15.432165 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.432113 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-ktvjm" Jul 23 18:17:15.624496 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.624454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:15.724128 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.724091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:15.915916 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:15.915873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:16.280211 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:16.280168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:17.346605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:17.346561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:17.404819 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:17.404787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:17.497425 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:17.497387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:17.598052 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:17.597975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:18.517754 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.517722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:18.577541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.577500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:18.605805 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.605774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:17:18.605939 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.605913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:17:18.609408 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.609383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 18:17:18.609536 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.609442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 18:17:18.618815 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.618796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17732286-b41e-4c5d-913a-440a7677d70d-metrics-certs\") pod \"network-metrics-daemon-wc7ll\" (UID: \"17732286-b41e-4c5d-913a-440a7677d70d\") " pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:17:18.618927 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.618840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b31242ed-e9d8-4a85-b061-2b226f509c93-original-pull-secret\") pod \"global-pull-secret-syncer-wdlln\" (UID: \"b31242ed-e9d8-4a85-b061-2b226f509c93\") " pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:17:18.706600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.706565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:17:18.709331 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.709292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzks\" (UniqueName: \"kubernetes.io/projected/38288d42-be78-49db-908d-a5d10bbca786-kube-api-access-xdzks\") pod \"network-check-target-g7glb\" (UID: \"38288d42-be78-49db-908d-a5d10bbca786\") " pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:17:18.743489 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.743464 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w7wd2\"" Jul 23 18:17:18.748334 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.748302 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdlln" Jul 23 18:17:18.751406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.751378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:17:18.755963 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.755943 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsfk8\"" Jul 23 18:17:18.765013 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:18.764990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wc7ll" Jul 23 18:17:20.023767 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.023742 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wc7ll"] Jul 23 18:17:20.028051 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:20.028010 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17732286_b41e_4c5d_913a_440a7677d70d.slice/crio-80cfbc4a6b9462b83bf9945e7f32d77fe6a18193330ad7b74581bdaf75ef1658 WatchSource:0}: Error finding container 80cfbc4a6b9462b83bf9945e7f32d77fe6a18193330ad7b74581bdaf75ef1658: Status 404 returned error can't find the container with id 80cfbc4a6b9462b83bf9945e7f32d77fe6a18193330ad7b74581bdaf75ef1658 Jul 23 18:17:20.259674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.259590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n6f" event={"ID":"1bd220f8-70f0-491d-b362-cb4d9db633f4","Type":"ContainerStarted","Data":"41b6bee7614e89e10f69b97094ea9a83e75931d1c8173a26ee68f2bc74a20980"} Jul 23 18:17:20.259933 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.259907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7glb"] Jul 23 18:17:20.261066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.261033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdlln"] Jul 23 18:17:20.261122 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.261098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc7ll" event={"ID":"17732286-b41e-4c5d-913a-440a7677d70d","Type":"ContainerStarted","Data":"80cfbc4a6b9462b83bf9945e7f32d77fe6a18193330ad7b74581bdaf75ef1658"} Jul 23 18:17:20.262992 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:20.262969 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38288d42_be78_49db_908d_a5d10bbca786.slice/crio-d44d65f2905c4b6ca483e7d1432db46b745e5156121a228659a997e7224388b4 WatchSource:0}: Error finding container d44d65f2905c4b6ca483e7d1432db46b745e5156121a228659a997e7224388b4: Status 404 returned error can't find the container with id d44d65f2905c4b6ca483e7d1432db46b745e5156121a228659a997e7224388b4 Jul 23 18:17:20.263429 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:20.263412 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31242ed_e9d8_4a85_b061_2b226f509c93.slice/crio-d5e8da493c8f216765bbf01726ce4092ddda641bc66c295491884dad18527245 WatchSource:0}: Error finding container d5e8da493c8f216765bbf01726ce4092ddda641bc66c295491884dad18527245: Status 404 returned error can't find the container with id d5e8da493c8f216765bbf01726ce4092ddda641bc66c295491884dad18527245 Jul 23 18:17:20.301593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.301564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:20.400702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:20.400620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:21.122870 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.122762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:21.125783 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.125744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4bc9a111-88cb-4371-a4b7-2a8161730ec4-metrics-tls\") pod \"dns-default-rhlsn\" (UID: \"4bc9a111-88cb-4371-a4b7-2a8161730ec4\") " pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:21.266643 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.266569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdlln" event={"ID":"b31242ed-e9d8-4a85-b061-2b226f509c93","Type":"ContainerStarted","Data":"d5e8da493c8f216765bbf01726ce4092ddda641bc66c295491884dad18527245"} Jul 23 18:17:21.268800 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.268768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7glb" event={"ID":"38288d42-be78-49db-908d-a5d10bbca786","Type":"ContainerStarted","Data":"a408789ac6695efda525563db4f35f8ee83a10601665fbad077c497f3bfaa216"} Jul 23 18:17:21.268933 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.268819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7glb" event={"ID":"38288d42-be78-49db-908d-a5d10bbca786","Type":"ContainerStarted","Data":"d44d65f2905c4b6ca483e7d1432db46b745e5156121a228659a997e7224388b4"} Jul 23 18:17:21.268933 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.268902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:17:21.283266 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.283166 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g7glb" podStartSLOduration=67.283152268 podStartE2EDuration="1m7.283152268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:21.282799243 +0000 UTC m=+68.013474334" watchObservedRunningTime="2026-07-23 18:17:21.283152268 +0000 UTC m=+68.013827358" Jul 23 18:17:21.305502 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.305481 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w4ck5\"" Jul 23 18:17:21.313172 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.313156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:21.478882 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:21.478810 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rhlsn"] Jul 23 18:17:21.484582 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:21.483822 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc9a111_88cb_4371_a4b7_2a8161730ec4.slice/crio-bf4a7512c1ecc8b410caf512b0563fbaa0ad91800e42a421b6a077e3527a3cb5 WatchSource:0}: Error finding container bf4a7512c1ecc8b410caf512b0563fbaa0ad91800e42a421b6a077e3527a3cb5: Status 404 returned error can't find the container with id bf4a7512c1ecc8b410caf512b0563fbaa0ad91800e42a421b6a077e3527a3cb5 Jul 23 18:17:22.274729 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:22.274653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhlsn" event={"ID":"4bc9a111-88cb-4371-a4b7-2a8161730ec4","Type":"ContainerStarted","Data":"bf4a7512c1ecc8b410caf512b0563fbaa0ad91800e42a421b6a077e3527a3cb5"} Jul 23 18:17:22.277244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:22.277193 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc7ll" event={"ID":"17732286-b41e-4c5d-913a-440a7677d70d","Type":"ContainerStarted","Data":"cd6ccec3870e21a43c9800285c3e470b2952b8f7d98bbe43e5741f30dc336bb3"} Jul 23 18:17:22.277382 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:22.277268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wc7ll" event={"ID":"17732286-b41e-4c5d-913a-440a7677d70d","Type":"ContainerStarted","Data":"02d875ce3f891c926f8ff45651c0f8f6d104f1e817d9545a1b62bd167186131a"} Jul 23 18:17:22.293796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:22.293747 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wc7ll" podStartSLOduration=67.008656109 podStartE2EDuration="1m8.293731614s" totalImagesPullingTime="1.285075505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:22.292187517 +0000 UTC m=+69.022862618" watchObservedRunningTime="2026-07-23 18:17:22.293731614 +0000 UTC m=+69.024406755" Jul 23 18:17:23.294106 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.294075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg"] Jul 23 18:17:23.297544 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.297503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.299984 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.299959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dbzz9\"" Jul 23 18:17:23.300284 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.300266 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 18:17:23.301040 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.301016 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 18:17:23.301674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.301289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 18:17:23.301674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.301502 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 18:17:23.313815 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.313751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg"] Jul 23 18:17:23.322162 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.319461 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:17:23.343945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.343355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c65e1e40-e205-437d-962c-ff55add98809-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.343945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.343446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtpw\" (UniqueName: \"kubernetes.io/projected/c65e1e40-e205-437d-962c-ff55add98809-kube-api-access-vjtpw\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.353839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.353748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:23.393188 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.393160 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-fk5ck"] Jul 23 18:17:23.396962 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.396943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:23.397073 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.397056 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56d5b57745-gcm5f"] Jul 23 18:17:23.399762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.399532 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 18:17:23.399762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.399571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 18:17:23.399762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.399631 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.400545 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.400380 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wknw6\"" Jul 23 18:17:23.418103 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.417751 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-fk5ck"] Jul 23 18:17:23.419612 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.419595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d5b57745-gcm5f"] Jul 23 18:17:23.444163 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64cb1b91-c587-4298-94d7-c19c0a495986-ca-trust-extracted\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtpw\" (UniqueName: \"kubernetes.io/projected/c65e1e40-e205-437d-962c-ff55add98809-kube-api-access-vjtpw\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.444304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjm8\" (UniqueName: \"kubernetes.io/projected/1b4b3795-c27a-4228-a220-d856254e06dc-kube-api-access-xxjm8\") pod \"downloads-5bc67dcd46-fk5ck\" (UID: \"1b4b3795-c27a-4228-a220-d856254e06dc\") " pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:23.444304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-bound-sa-token\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-registry-certificates\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-installation-pull-secrets\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-registry-tls\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c65e1e40-e205-437d-962c-ff55add98809-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-trusted-ca\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dr8\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-kube-api-access-d7dr8\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.444593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.444508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-image-registry-private-configuration\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.447377 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.447344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c65e1e40-e205-437d-962c-ff55add98809-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.460103 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.460080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtpw\" (UniqueName: \"kubernetes.io/projected/c65e1e40-e205-437d-962c-ff55add98809-kube-api-access-vjtpw\") pod \"managed-serviceaccount-addon-agent-66c74c97cd-7f8xg\" (UID: \"c65e1e40-e205-437d-962c-ff55add98809\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.545308 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-registry-tls\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545308 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-trusted-ca\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dr8\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-kube-api-access-d7dr8\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-image-registry-private-configuration\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64cb1b91-c587-4298-94d7-c19c0a495986-ca-trust-extracted\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjm8\" (UniqueName: \"kubernetes.io/projected/1b4b3795-c27a-4228-a220-d856254e06dc-kube-api-access-xxjm8\") pod \"downloads-5bc67dcd46-fk5ck\" (UID: \"1b4b3795-c27a-4228-a220-d856254e06dc\") " pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-bound-sa-token\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-registry-certificates\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.545538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.545514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-installation-pull-secrets\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.546510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.546479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64cb1b91-c587-4298-94d7-c19c0a495986-ca-trust-extracted\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.546665 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.546644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-trusted-ca\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.547039 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.547015 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64cb1b91-c587-4298-94d7-c19c0a495986-registry-certificates\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.548053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.548010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-image-registry-private-configuration\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.548240 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.548204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-registry-tls\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.548322 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.548245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64cb1b91-c587-4298-94d7-c19c0a495986-installation-pull-secrets\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.555056 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.555034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-bound-sa-token\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.555375 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.555358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dr8\" (UniqueName: \"kubernetes.io/projected/64cb1b91-c587-4298-94d7-c19c0a495986-kube-api-access-d7dr8\") pod \"image-registry-56d5b57745-gcm5f\" (UID: \"64cb1b91-c587-4298-94d7-c19c0a495986\") " pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:23.556104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.556081 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjm8\" (UniqueName: \"kubernetes.io/projected/1b4b3795-c27a-4228-a220-d856254e06dc-kube-api-access-xxjm8\") pod \"downloads-5bc67dcd46-fk5ck\" (UID: \"1b4b3795-c27a-4228-a220-d856254e06dc\") " pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:23.618306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.618065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" Jul 23 18:17:23.710447 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.710414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:23.719298 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:23.719274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:24.404440 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.404359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:24.466660 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.466624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:24.527940 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.527908 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:17:24.530516 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.530489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.532897 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.532872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnxx6\"" Jul 23 18:17:24.533036 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.532984 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 18:17:24.533702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.533683 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 18:17:24.533807 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.533784 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 18:17:24.533923 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.533904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 18:17:24.534031 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.533937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 18:17:24.542721 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.542700 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:17:24.653749 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653719 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.653928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.653928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.653928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653849 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qr67\" (UniqueName: \"kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.653928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.653928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.653916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.704744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.704708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:24.755357 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qr67\" (UniqueName: \"kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.755508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.755511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.756161 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.756132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.756278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.756132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.756278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.756184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.758583 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.758532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.758583 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.758544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.758706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.758583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/45587963-6805-4d17-b098-64a886a354cc-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-n8gtg\" (UID: \"45587963-6805-4d17-b098-64a886a354cc\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:17:24.760186 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.760159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:24.763397 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.763372 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qr67\" (UniqueName: \"kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67\") pod \"console-9444d655d-pwxqg\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.841775 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.841750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:24.937717 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.937692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f6ntr\"" Jul 23 18:17:24.946620 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:24.946596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" Jul 23 18:17:25.928508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:25.928414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg"] Jul 23 18:17:25.983703 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:25.983669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d5b57745-gcm5f"] Jul 23 18:17:26.000648 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:26.000617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb1b91_c587_4298_94d7_c19c0a495986.slice/crio-b4b15d7f4154ffe289c2348470a83137053b64ef48fab74a40830c228887c22e WatchSource:0}: Error finding container b4b15d7f4154ffe289c2348470a83137053b64ef48fab74a40830c228887c22e: Status 404 returned error can't find the container with id b4b15d7f4154ffe289c2348470a83137053b64ef48fab74a40830c228887c22e Jul 23 18:17:26.195604 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.195574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg"] Jul 23 18:17:26.198363 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:26.198336 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc65e1e40_e205_437d_962c_ff55add98809.slice/crio-f1f54bf07d93995c0de022b1a1e85135699edd0d882483e31f8045d1cf5d3fbc WatchSource:0}: Error finding container f1f54bf07d93995c0de022b1a1e85135699edd0d882483e31f8045d1cf5d3fbc: Status 404 returned error can't find the container with id f1f54bf07d93995c0de022b1a1e85135699edd0d882483e31f8045d1cf5d3fbc Jul 23 18:17:26.202465 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.202442 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:17:26.204803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.204780 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-fk5ck"] Jul 23 18:17:26.204942 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:26.204925 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdc119e_34c0_4c59_a2ca_236ccda620d7.slice/crio-340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998 WatchSource:0}: Error finding container 340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998: Status 404 returned error can't find the container with id 340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998 Jul 23 18:17:26.207721 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:26.207695 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b3795_c27a_4228_a220_d856254e06dc.slice/crio-326bd33536885d6561fe70678ecbeabcad8f53215a9cbc6ea04df31400a3c98e WatchSource:0}: Error finding container 326bd33536885d6561fe70678ecbeabcad8f53215a9cbc6ea04df31400a3c98e: Status 404 returned error can't find the container with id 326bd33536885d6561fe70678ecbeabcad8f53215a9cbc6ea04df31400a3c98e Jul 23 18:17:26.298906 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.298866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n6f" event={"ID":"1bd220f8-70f0-491d-b362-cb4d9db633f4","Type":"ContainerStarted","Data":"e5deeb83ebfff92a12caae7f9925774b2b486874fa3ce1821f3e427ab4eb7a34"} Jul 23 18:17:26.300582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.300549 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdlln" event={"ID":"b31242ed-e9d8-4a85-b061-2b226f509c93","Type":"ContainerStarted","Data":"e4299f0db1f00addc95905e59863ce758bc9b9e5756dfc48992022462e533f11"} Jul 23 18:17:26.301942 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.301913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-fk5ck" event={"ID":"1b4b3795-c27a-4228-a220-d856254e06dc","Type":"ContainerStarted","Data":"326bd33536885d6561fe70678ecbeabcad8f53215a9cbc6ea04df31400a3c98e"} Jul 23 18:17:26.303343 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.303316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9444d655d-pwxqg" event={"ID":"2bdc119e-34c0-4c59-a2ca-236ccda620d7","Type":"ContainerStarted","Data":"340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998"} Jul 23 18:17:26.304569 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.304544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" event={"ID":"c65e1e40-e205-437d-962c-ff55add98809","Type":"ContainerStarted","Data":"f1f54bf07d93995c0de022b1a1e85135699edd0d882483e31f8045d1cf5d3fbc"} Jul 23 18:17:26.306286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.306257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" event={"ID":"64cb1b91-c587-4298-94d7-c19c0a495986","Type":"ContainerStarted","Data":"ad22e8f31388df9c1f151925a706ba5e3676b9f1a82ce2f5af5f7b19f288a123"} Jul 23 18:17:26.306394 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.306301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" event={"ID":"64cb1b91-c587-4298-94d7-c19c0a495986","Type":"ContainerStarted","Data":"b4b15d7f4154ffe289c2348470a83137053b64ef48fab74a40830c228887c22e"} Jul 23 18:17:26.306394 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.306374 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:26.308505 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.308485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhlsn" event={"ID":"4bc9a111-88cb-4371-a4b7-2a8161730ec4","Type":"ContainerStarted","Data":"873b172aa4e52379e3eb6f0ecdc77a1a21271df322c5621ff72f608bcfd1047d"} Jul 23 18:17:26.308610 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.308527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rhlsn" event={"ID":"4bc9a111-88cb-4371-a4b7-2a8161730ec4","Type":"ContainerStarted","Data":"bffdb368c56ae756c0e9b4fd0476650562b766cd61732107f592f8b2025b9a23"} Jul 23 18:17:26.308610 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.308574 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:26.310013 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.309990 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" event={"ID":"45587963-6805-4d17-b098-64a886a354cc","Type":"ContainerStarted","Data":"8b16d6a07612a2ed50a269f8f988db5f91b8ce45d9014350ef4e5f22c4f6851e"} Jul 23 18:17:26.316345 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.316305 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wdlln" podStartSLOduration=66.908230912 podStartE2EDuration="1m12.316294103s" totalImagesPullingTime="5.408063191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:26.315248779 +0000 UTC m=+73.045923888" watchObservedRunningTime="2026-07-23 18:17:26.316294103 +0000 UTC m=+73.046969172" Jul 23 18:17:26.331911 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.331862 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rhlsn" podStartSLOduration=33.117834471 podStartE2EDuration="37.331849407s" totalImagesPullingTime="4.214014936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:26.330806761 +0000 UTC m=+73.061481852" watchObservedRunningTime="2026-07-23 18:17:26.331849407 +0000 UTC m=+73.062524498" Jul 23 18:17:26.349721 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:26.348663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" podStartSLOduration=3.348644767 podStartE2EDuration="3.348644767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:26.347810267 +0000 UTC m=+73.078485367" watchObservedRunningTime="2026-07-23 18:17:26.348644767 +0000 UTC m=+73.079319840" Jul 23 18:17:27.598759 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:27.598677 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:27.668915 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:27.668853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:28.327571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:28.327533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" event={"ID":"45587963-6805-4d17-b098-64a886a354cc","Type":"ContainerStarted","Data":"eb2af29206b5bf8100e88b52568bd83faff16c6bd425f98fa82e47356adc1730"} Jul 23 18:17:28.356662 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:28.355614 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-n8gtg" podStartSLOduration=34.418426503 podStartE2EDuration="36.35559438s" totalImagesPullingTime="1.937167877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:28.353198726 +0000 UTC m=+75.083873828" watchObservedRunningTime="2026-07-23 18:17:28.35559438 +0000 UTC m=+75.086269473" Jul 23 18:17:28.507588 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:28.507523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:28.610066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:28.609969 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:30.122806 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:30.122751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:30.183532 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:30.183500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.019337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.019302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.120252 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.120124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.276617 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.276537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.374476 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.374444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.450744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.450711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:31.677292 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:31.677201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:33.326439 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.326371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:33.351003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.350969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9444d655d-pwxqg" event={"ID":"2bdc119e-34c0-4c59-a2ca-236ccda620d7","Type":"ContainerStarted","Data":"834ba02132170fa1c1df8bfc5226a7abfb096ccb716d5cb563baf7af75f56727"} Jul 23 18:17:33.352790 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.352756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" event={"ID":"c65e1e40-e205-437d-962c-ff55add98809","Type":"ContainerStarted","Data":"235ba16e1ad181c0af6e067b6fd1b45ea3ae211826a1b16c5b03f5755a7ad6c1"} Jul 23 18:17:33.354754 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.354734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s4n6f" event={"ID":"1bd220f8-70f0-491d-b362-cb4d9db633f4","Type":"ContainerStarted","Data":"b56ad442957fe3ec0c942aca35556a64c5891f7bf588a673a8b4175b50422342"} Jul 23 18:17:33.367757 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.367719 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9444d655d-pwxqg" podStartSLOduration=2.439227432 podStartE2EDuration="9.367706931s" totalImagesPullingTime="6.928479499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:33.366954554 +0000 UTC m=+80.097629667" watchObservedRunningTime="2026-07-23 18:17:33.367706931 +0000 UTC m=+80.098382003" Jul 23 18:17:33.384671 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.384598 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s4n6f" podStartSLOduration=17.530680523 podStartE2EDuration="35.38459042s" totalImagesPullingTime="17.853909897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:16:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:33.383502836 +0000 UTC m=+80.114177927" watchObservedRunningTime="2026-07-23 18:17:33.38459042 +0000 UTC m=+80.115265511" Jul 23 18:17:33.398494 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:33.398459 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c74c97cd-7f8xg" podStartSLOduration=3.457058156 podStartE2EDuration="10.398446961s" totalImagesPullingTime="6.941388805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:33.397954612 +0000 UTC m=+80.128629704" watchObservedRunningTime="2026-07-23 18:17:33.398446961 +0000 UTC m=+80.129122052" Jul 23 18:17:34.019821 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.019786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:34.121321 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.121285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:34.181779 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.181740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:34.842814 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.842778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:34.843207 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.842870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:34.847966 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:34.847942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:35.367663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:35.367632 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:17:35.923033 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:35.923001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:36.023514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:36.023480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:36.331797 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:36.331765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rhlsn" Jul 23 18:17:37.338315 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:37.338276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:37.438446 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:37.438410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:40.139495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:40.139436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:40.201328 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:40.201285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:40.685250 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:40.685202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:40.786664 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:40.786629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:41.464781 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:41.464732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:41.540118 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:41.540063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:41.638928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:41.638887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:43.211919 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:43.211888 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:17:43.734974 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:43.734939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:43.840101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:43.840055 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:44.120917 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:44.120818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:44.221691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:44.221649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:45.306406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:45.306372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:45.696199 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:45.696159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:46.201009 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.200970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:46.301000 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.300920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:46.406309 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.406267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-fk5ck" event={"ID":"1b4b3795-c27a-4228-a220-d856254e06dc","Type":"ContainerStarted","Data":"3d0a918237347146eaf90dfe0f8e8b6432eb109b9b8cd098b684d98e0132ee3b"} Jul 23 18:17:46.406757 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.406490 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:46.413694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.413672 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-fk5ck" Jul 23 18:17:46.457528 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.457475 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-fk5ck" podStartSLOduration=3.708313266 podStartE2EDuration="23.457461692s" totalImagesPullingTime="19.749148426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:46.456299077 +0000 UTC m=+93.186974175" watchObservedRunningTime="2026-07-23 18:17:46.457461692 +0000 UTC m=+93.188136783" Jul 23 18:17:46.691276 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.691161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:46.793201 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:46.793165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:47.335941 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.335906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56d5b57745-gcm5f" Jul 23 18:17:47.482863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.482822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:47.598198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.598119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:47.684217 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.684175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:47.841909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.841870 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:47.883470 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.883385 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h9zq6"] Jul 23 18:17:47.914050 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.914013 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:47.916761 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.916719 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvvxz\"" Jul 23 18:17:47.916902 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.916807 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 18:17:47.917094 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.917074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 18:17:47.917190 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.917114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 18:17:47.917558 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.917529 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 18:17:47.917674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.917537 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 18:17:47.917674 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.917560 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 18:17:47.940065 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:47.940042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.018246 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.018193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.051239 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-wtmp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvtp\" (UniqueName: \"kubernetes.io/projected/fa0502df-2c99-4927-bdc7-687173c5a859-kube-api-access-dnvtp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-metrics-client-ca\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-sys\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-root\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-textfile\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.051636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.051569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-tls\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.122037 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.121996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.152155 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvtp\" (UniqueName: \"kubernetes.io/projected/fa0502df-2c99-4927-bdc7-687173c5a859-kube-api-access-dnvtp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152155 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-metrics-client-ca\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-sys\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-root\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-textfile\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-tls\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-wtmp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-root\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152814 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-metrics-client-ca\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152892 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-textfile\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-accelerators-collector-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.152967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152924 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-wtmp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.153132 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.152974 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa0502df-2c99-4927-bdc7-687173c5a859-sys\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.155201 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.155179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.155481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.155456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fa0502df-2c99-4927-bdc7-687173c5a859-node-exporter-tls\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.160390 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.160367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvtp\" (UniqueName: \"kubernetes.io/projected/fa0502df-2c99-4927-bdc7-687173c5a859-kube-api-access-dnvtp\") pod \"node-exporter-h9zq6\" (UID: \"fa0502df-2c99-4927-bdc7-687173c5a859\") " pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.225495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.225452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h9zq6" Jul 23 18:17:48.228202 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.228167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.268393 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:17:48.268359 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0502df_2c99_4927_bdc7_687173c5a859.slice/crio-23d44f3843504ce2e42a788c8932e21a97885df6abcd2ce010cf6f3577e042ae WatchSource:0}: Error finding container 23d44f3843504ce2e42a788c8932e21a97885df6abcd2ce010cf6f3577e042ae: Status 404 returned error can't find the container with id 23d44f3843504ce2e42a788c8932e21a97885df6abcd2ce010cf6f3577e042ae Jul 23 18:17:48.349698 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.349651 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" podUID="eee0b41e-0f41-473c-99e4-79e955613805" containerName="registry" containerID="cri-o://771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65" gracePeriod=30 Jul 23 18:17:48.416407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.416346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9zq6" event={"ID":"fa0502df-2c99-4927-bdc7-687173c5a859","Type":"ContainerStarted","Data":"23d44f3843504ce2e42a788c8932e21a97885df6abcd2ce010cf6f3577e042ae"} Jul 23 18:17:48.453043 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.453012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.554416 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.553955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.616733 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.616697 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:48.757902 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.757865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.757935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.757973 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.757998 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.758026 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g52mh\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.758053 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.758078 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758418 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.758105 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration\") pod \"eee0b41e-0f41-473c-99e4-79e955613805\" (UID: \"eee0b41e-0f41-473c-99e4-79e955613805\") " Jul 23 18:17:48.758538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.758507 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:17:48.759198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.759154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:17:48.761786 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.761745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:17:48.762067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.762039 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:17:48.763783 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.763757 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:17:48.763932 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.763915 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh" (OuterVolumeSpecName: "kube-api-access-g52mh") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "kube-api-access-g52mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:17:48.764162 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.764135 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:17:48.770176 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.770124 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "eee0b41e-0f41-473c-99e4-79e955613805" (UID: "eee0b41e-0f41-473c-99e4-79e955613805"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:17:48.831300 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.831168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:48.859314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859285 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eee0b41e-0f41-473c-99e4-79e955613805-ca-trust-extracted\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859311 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-trusted-ca\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859325 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-image-registry-private-configuration\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859339 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eee0b41e-0f41-473c-99e4-79e955613805-registry-certificates\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859350 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-bound-sa-token\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859361 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eee0b41e-0f41-473c-99e4-79e955613805-installation-pull-secrets\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859373 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-registry-tls\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.859701 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.859386 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52mh\" (UniqueName: \"kubernetes.io/projected/eee0b41e-0f41-473c-99e4-79e955613805-kube-api-access-g52mh\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:17:48.925872 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:48.925765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:49.422260 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.422158 2576 generic.go:356] "Generic (PLEG): container finished" podID="eee0b41e-0f41-473c-99e4-79e955613805" containerID="771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65" exitCode=0 Jul 23 18:17:49.422260 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.422213 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" Jul 23 18:17:49.422566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.422257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" event={"ID":"eee0b41e-0f41-473c-99e4-79e955613805","Type":"ContainerDied","Data":"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65"} Jul 23 18:17:49.422566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.422321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75fbf98c45-xhghl" event={"ID":"eee0b41e-0f41-473c-99e4-79e955613805","Type":"ContainerDied","Data":"b9cd8e053a7763af140eea79a873ea6bbfd4c8908b680495c7f8959ecde8565f"} Jul 23 18:17:49.422566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.422355 2576 scope.go:117] "RemoveContainer" containerID="771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65" Jul 23 18:17:49.446782 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.446749 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:17:49.453161 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.453137 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75fbf98c45-xhghl"] Jul 23 18:17:49.588529 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.588500 2576 scope.go:117] "RemoveContainer" containerID="771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65" Jul 23 18:17:49.588946 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:17:49.588914 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65\": container with ID starting with 771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65 not found: ID does not exist" containerID="771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65" Jul 23 18:17:49.589075 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.588953 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65"} err="failed to get container status \"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65\": rpc error: code = NotFound desc = could not find container \"771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65\": container with ID starting with 771ee29b065f19094082c8d701208107b282139b25cedb6cef96aa08a208ff65 not found: ID does not exist" Jul 23 18:17:49.840993 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.840958 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee0b41e-0f41-473c-99e4-79e955613805" path="/var/lib/kubelet/pods/eee0b41e-0f41-473c-99e4-79e955613805/volumes" Jul 23 18:17:49.987134 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:49.987093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:50.278904 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.278825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:50.356084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.356036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:50.429851 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.429814 2576 generic.go:356] "Generic (PLEG): container finished" podID="fa0502df-2c99-4927-bdc7-687173c5a859" containerID="a525cf0c5df2ea4b366c3845935721158e224c76a8a04d8eb29fdf9de076d92a" exitCode=0 Jul 23 18:17:50.430068 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.429874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9zq6" event={"ID":"fa0502df-2c99-4927-bdc7-687173c5a859","Type":"ContainerDied","Data":"a525cf0c5df2ea4b366c3845935721158e224c76a8a04d8eb29fdf9de076d92a"} Jul 23 18:17:50.792694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.792658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:50.882751 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.882673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:50.981972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:50.981914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:51.046678 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:51.046638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:51.436126 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:51.436093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9zq6" event={"ID":"fa0502df-2c99-4927-bdc7-687173c5a859","Type":"ContainerStarted","Data":"b0ed7896ce591f7b0fb74eb1023717c173ae056c7407ec587eeea23432ea6a57"} Jul 23 18:17:51.436342 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:51.436153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h9zq6" event={"ID":"fa0502df-2c99-4927-bdc7-687173c5a859","Type":"ContainerStarted","Data":"c20e61e24726a83c02b8724eaa1bc9017363c460f400767b58ac051bc1a93d77"} Jul 23 18:17:51.461610 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:51.461549 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h9zq6" podStartSLOduration=3.043060295 podStartE2EDuration="4.461528275s" totalImagesPullingTime="1.322031189s" totalInitContainerRuntime="96.436791ms" isStatefulPod=true podCreationTimestamp="2026-07-23 18:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:17:51.460011048 +0000 UTC m=+98.190686140" watchObservedRunningTime="2026-07-23 18:17:51.461528275 +0000 UTC m=+98.192203491" Jul 23 18:17:52.280518 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:52.280489 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g7glb" Jul 23 18:17:52.681170 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:52.681085 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:52.771836 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:52.771799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:52.871881 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:52.871852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:52.972259 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:52.972207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:54.199520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:54.199483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:54.528543 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:54.528459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:54.596608 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:54.596572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:58.449030 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:58.448993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:17:58.575585 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:17:58.575544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:03.505869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:03.505836 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:03.595891 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:03.595859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:03.694590 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:03.694549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:03.793115 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:03.793021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:05.282663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:05.282631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:05.384738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:05.384700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:08.241074 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.241003 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9444d655d-pwxqg" podUID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" containerName="console" containerID="cri-o://834ba02132170fa1c1df8bfc5226a7abfb096ccb716d5cb563baf7af75f56727" gracePeriod=15 Jul 23 18:18:08.497840 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.497756 2576 generic.go:356] "Generic (PLEG): container finished" podID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" containerID="834ba02132170fa1c1df8bfc5226a7abfb096ccb716d5cb563baf7af75f56727" exitCode=0 Jul 23 18:18:08.497840 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.497818 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9444d655d-pwxqg" event={"ID":"2bdc119e-34c0-4c59-a2ca-236ccda620d7","Type":"ContainerDied","Data":"834ba02132170fa1c1df8bfc5226a7abfb096ccb716d5cb563baf7af75f56727"} Jul 23 18:18:08.498011 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.497857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9444d655d-pwxqg" event={"ID":"2bdc119e-34c0-4c59-a2ca-236ccda620d7","Type":"ContainerDied","Data":"340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998"} Jul 23 18:18:08.498011 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.497870 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340c7df6cd4cc241ddb57c9edb599299b4b8a74b2bb6eb79c169ed86eec0c998" Jul 23 18:18:08.517556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.517528 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:18:08.605101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605052 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qr67\" (UniqueName: \"kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605158 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605218 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert\") pod \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\" (UID: \"2bdc119e-34c0-4c59-a2ca-236ccda620d7\") " Jul 23 18:18:08.605685 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605644 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config" (OuterVolumeSpecName: "console-config") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:18:08.605817 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca" (OuterVolumeSpecName: "service-ca") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:18:08.605817 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.605764 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:18:08.607540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.607514 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:18:08.607669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.607569 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67" (OuterVolumeSpecName: "kube-api-access-5qr67") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "kube-api-access-5qr67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:18:08.607669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.607611 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2bdc119e-34c0-4c59-a2ca-236ccda620d7" (UID: "2bdc119e-34c0-4c59-a2ca-236ccda620d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:18:08.706167 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706125 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-service-ca\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:08.706167 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706164 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-oauth-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:08.706167 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706176 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:08.706446 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706189 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qr67\" (UniqueName: \"kubernetes.io/projected/2bdc119e-34c0-4c59-a2ca-236ccda620d7-kube-api-access-5qr67\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:08.706446 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706201 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-oauth-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:08.706446 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:08.706213 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdc119e-34c0-4c59-a2ca-236ccda620d7-console-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:18:09.501259 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:09.501212 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9444d655d-pwxqg" Jul 23 18:18:09.521407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:09.521378 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:18:09.528433 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:09.528408 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9444d655d-pwxqg"] Jul 23 18:18:09.833839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:09.833764 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" path="/var/lib/kubelet/pods/2bdc119e-34c0-4c59-a2ca-236ccda620d7/volumes" Jul 23 18:18:11.701215 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:11.701174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:13.239350 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:13.239305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:13.518290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:13.518187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:29.100189 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:29.100132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:29.158958 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:29.158925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:33.949768 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:33.949729 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:34.006497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:34.006461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:36.020093 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:36.020062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:18:41.479023 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:18:41.478982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:07.099327 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:07.099248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:07.230948 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:07.230916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:16.283020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.282985 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283267 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee0b41e-0f41-473c-99e4-79e955613805" containerName="registry" Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283279 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee0b41e-0f41-473c-99e4-79e955613805" containerName="registry" Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283288 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" containerName="console" Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283293 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" containerName="console" Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283348 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee0b41e-0f41-473c-99e4-79e955613805" containerName="registry" Jul 23 18:19:16.283482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.283358 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdc119e-34c0-4c59-a2ca-236ccda620d7" containerName="console" Jul 23 18:19:16.285211 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.285125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.287400 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.287379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 18:19:16.288456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.288402 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 18:19:16.288456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.288417 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnxx6\"" Jul 23 18:19:16.288619 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.288500 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 18:19:16.288619 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.288505 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 18:19:16.288619 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.288543 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 18:19:16.293117 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.292958 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 18:19:16.301286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.301264 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:16.350807 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.350776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:16.398838 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.398814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.398935 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.398847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.398935 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.398868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.403110 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.398887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.403110 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.399415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.403110 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.399478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnkb\" (UniqueName: \"kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.403110 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.399533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.442978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.442956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:16.499941 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.499916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500036 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.499951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500036 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.499973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500036 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500179 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500179 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnkb\" (UniqueName: \"kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500179 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500778 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.500957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.500934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.502481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.502450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.502565 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.502532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.507455 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.507435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnkb\" (UniqueName: \"kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb\") pod \"console-5bc768d9c7-g8krg\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.595826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.595760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:16.738090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:16.738043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:16.740269 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:19:16.740236 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc0ab7e_054b_43d2_bbe7_f463a98dd456.slice/crio-be36ff89811bbff0c09e0b6f8a4b14ff45c5c816ffae7dbd6cb07ab8d53b0738 WatchSource:0}: Error finding container be36ff89811bbff0c09e0b6f8a4b14ff45c5c816ffae7dbd6cb07ab8d53b0738: Status 404 returned error can't find the container with id be36ff89811bbff0c09e0b6f8a4b14ff45c5c816ffae7dbd6cb07ab8d53b0738 Jul 23 18:19:17.745249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:17.745194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc768d9c7-g8krg" event={"ID":"5fc0ab7e-054b-43d2-bbe7-f463a98dd456","Type":"ContainerStarted","Data":"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2"} Jul 23 18:19:17.745694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:17.745259 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc768d9c7-g8krg" event={"ID":"5fc0ab7e-054b-43d2-bbe7-f463a98dd456","Type":"ContainerStarted","Data":"be36ff89811bbff0c09e0b6f8a4b14ff45c5c816ffae7dbd6cb07ab8d53b0738"} Jul 23 18:19:17.762546 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:17.762490 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc768d9c7-g8krg" podStartSLOduration=1.762473928 podStartE2EDuration="1.762473928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:19:17.761652711 +0000 UTC m=+184.492327831" watchObservedRunningTime="2026-07-23 18:19:17.762473928 +0000 UTC m=+184.493149019" Jul 23 18:19:22.332022 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:22.331985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:22.433366 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:22.433321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:25.955970 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:25.955940 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:25.997502 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:25.997474 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:19:25.999564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:25.999547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.010050 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.010030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:19:26.064310 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wctf\" (UniqueName: \"kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064631 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.064631 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.064516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.164933 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.164900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.164933 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.164937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wctf\" (UniqueName: \"kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165139 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165195 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165195 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165334 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165334 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165799 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165773 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165799 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.165995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.165862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.166059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.166019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.167602 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.167572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.167757 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.167737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.172668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.172649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wctf\" (UniqueName: \"kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf\") pod \"console-69d9f9df64-p5wzn\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.308902 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.308829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:26.435096 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.435064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:19:26.438387 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:19:26.438361 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6883de29_c53a_4f88_a2bc_bef51555282b.slice/crio-dad2497a47e47c7eed63f8d370c66320fc7adb7e41c2c752006ae2a8120f78f0 WatchSource:0}: Error finding container dad2497a47e47c7eed63f8d370c66320fc7adb7e41c2c752006ae2a8120f78f0: Status 404 returned error can't find the container with id dad2497a47e47c7eed63f8d370c66320fc7adb7e41c2c752006ae2a8120f78f0 Jul 23 18:19:26.596612 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.596546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:26.778022 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.777985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d9f9df64-p5wzn" event={"ID":"6883de29-c53a-4f88-a2bc-bef51555282b","Type":"ContainerStarted","Data":"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1"} Jul 23 18:19:26.778205 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.778034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d9f9df64-p5wzn" event={"ID":"6883de29-c53a-4f88-a2bc-bef51555282b","Type":"ContainerStarted","Data":"dad2497a47e47c7eed63f8d370c66320fc7adb7e41c2c752006ae2a8120f78f0"} Jul 23 18:19:26.794201 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:26.794153 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69d9f9df64-p5wzn" podStartSLOduration=1.7941379610000001 podStartE2EDuration="1.794137961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:19:26.793863713 +0000 UTC m=+193.524538842" watchObservedRunningTime="2026-07-23 18:19:26.794137961 +0000 UTC m=+193.524813052" Jul 23 18:19:34.264132 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:34.264099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:34.320801 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:34.320769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:36.309820 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:36.309787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:36.310198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:36.309877 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:36.314784 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:36.314766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:36.821742 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:36.821714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:19:36.964189 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:36.964158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:37.067495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:37.067459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:50.288843 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:50.288807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:50.388159 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:50.388127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:19:50.977403 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:50.977345 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bc768d9c7-g8krg" podUID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" containerName="console" containerID="cri-o://ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2" gracePeriod=15 Jul 23 18:19:51.220413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.220390 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:51.341663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341583 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.341663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.341663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjnkb\" (UniqueName: \"kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341671 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341716 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert\") pod \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\" (UID: \"5fc0ab7e-054b-43d2-bbe7-f463a98dd456\") " Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.341954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca" (OuterVolumeSpecName: "service-ca") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.342188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:19:51.342249 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.342211 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:19:51.342510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.342271 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config" (OuterVolumeSpecName: "console-config") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:19:51.343789 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.343769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:19:51.343875 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.343855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb" (OuterVolumeSpecName: "kube-api-access-kjnkb") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "kube-api-access-kjnkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:19:51.343915 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.343878 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5fc0ab7e-054b-43d2-bbe7-f463a98dd456" (UID: "5fc0ab7e-054b-43d2-bbe7-f463a98dd456"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:19:51.442461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442427 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442457 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-service-ca\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442466 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjnkb\" (UniqueName: \"kubernetes.io/projected/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-kube-api-access-kjnkb\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442476 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-trusted-ca-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442486 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-oauth-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442495 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-console-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.442691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.442503 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fc0ab7e-054b-43d2-bbe7-f463a98dd456-oauth-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:19:51.870139 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.870099 2576 generic.go:356] "Generic (PLEG): container finished" podID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" containerID="ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2" exitCode=0 Jul 23 18:19:51.870324 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.870148 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc768d9c7-g8krg" Jul 23 18:19:51.870324 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.870163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc768d9c7-g8krg" event={"ID":"5fc0ab7e-054b-43d2-bbe7-f463a98dd456","Type":"ContainerDied","Data":"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2"} Jul 23 18:19:51.870324 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.870213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc768d9c7-g8krg" event={"ID":"5fc0ab7e-054b-43d2-bbe7-f463a98dd456","Type":"ContainerDied","Data":"be36ff89811bbff0c09e0b6f8a4b14ff45c5c816ffae7dbd6cb07ab8d53b0738"} Jul 23 18:19:51.870324 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.870255 2576 scope.go:117] "RemoveContainer" containerID="ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2" Jul 23 18:19:51.878278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.878257 2576 scope.go:117] "RemoveContainer" containerID="ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2" Jul 23 18:19:51.878544 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:19:51.878521 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2\": container with ID starting with ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2 not found: ID does not exist" containerID="ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2" Jul 23 18:19:51.878640 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.878549 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2"} err="failed to get container status \"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2\": rpc error: code = NotFound desc = could not find container \"ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2\": container with ID starting with ce6a965ffc5a3a029ff35fcd8749ea7f02756f3e8b2054e5d737a8dc005c59f2 not found: ID does not exist" Jul 23 18:19:51.885516 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.885484 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:51.888947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:51.888927 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc768d9c7-g8krg"] Jul 23 18:19:53.832475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:19:53.832438 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" path="/var/lib/kubelet/pods/5fc0ab7e-054b-43d2-bbe7-f463a98dd456/volumes" Jul 23 18:20:04.058020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:04.057982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:04.116744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:04.116708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:25.380049 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:25.380008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:25.492353 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:25.492321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:26.138659 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.138627 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r"] Jul 23 18:20:26.138883 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.138870 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" containerName="console" Jul 23 18:20:26.138883 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.138884 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" containerName="console" Jul 23 18:20:26.138971 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.138935 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fc0ab7e-054b-43d2-bbe7-f463a98dd456" containerName="console" Jul 23 18:20:26.141358 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.141341 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.143727 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.143707 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:20:26.144489 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.144469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:20:26.144537 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.144523 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5jb4d\"" Jul 23 18:20:26.152210 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.152190 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r"] Jul 23 18:20:26.289124 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.289090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.289268 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.289144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.289268 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.289205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.389690 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.389623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.389690 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.389653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.390032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.389692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.390032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.389958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.390109 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.390035 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.397787 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.397769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.450729 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.450693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:26.569543 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.569515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r"] Jul 23 18:20:26.989801 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:26.989767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" event={"ID":"8e76bd34-6d75-4344-b1e1-c83dd77c925c","Type":"ContainerStarted","Data":"d4592c3af6432589e3c8f0931b31f9e79049bf70f0e6458bd0c02e718d0aba77"} Jul 23 18:20:36.026066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:36.026034 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerID="67104f0bd230bd1d226eabed7fac7dd0fd17dcace11a67efeb5c297613cadc20" exitCode=0 Jul 23 18:20:36.026424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:36.026096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" event={"ID":"8e76bd34-6d75-4344-b1e1-c83dd77c925c","Type":"ContainerDied","Data":"67104f0bd230bd1d226eabed7fac7dd0fd17dcace11a67efeb5c297613cadc20"} Jul 23 18:20:38.036156 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:38.036122 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerID="74fb716f04821cfb53aa4eaf803642e41b594c8fbbff0be12ce9bbcd225682ac" exitCode=0 Jul 23 18:20:38.036510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:38.036195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" event={"ID":"8e76bd34-6d75-4344-b1e1-c83dd77c925c","Type":"ContainerDied","Data":"74fb716f04821cfb53aa4eaf803642e41b594c8fbbff0be12ce9bbcd225682ac"} Jul 23 18:20:46.068540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:46.068476 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerID="fb629f10e29c08b9e51292010aa21aeddeae407b0b14f7ade6a84780d0dc90ae" exitCode=0 Jul 23 18:20:46.068540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:46.068516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" event={"ID":"8e76bd34-6d75-4344-b1e1-c83dd77c925c","Type":"ContainerDied","Data":"fb629f10e29c08b9e51292010aa21aeddeae407b0b14f7ade6a84780d0dc90ae"} Jul 23 18:20:47.201504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.201483 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:47.348098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.348037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle\") pod \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " Jul 23 18:20:47.348098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.348075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util\") pod \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " Jul 23 18:20:47.348264 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.348120 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d\") pod \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\" (UID: \"8e76bd34-6d75-4344-b1e1-c83dd77c925c\") " Jul 23 18:20:47.348618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.348590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle" (OuterVolumeSpecName: "bundle") pod "8e76bd34-6d75-4344-b1e1-c83dd77c925c" (UID: "8e76bd34-6d75-4344-b1e1-c83dd77c925c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:47.350327 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.350297 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d" (OuterVolumeSpecName: "kube-api-access-g2s7d") pod "8e76bd34-6d75-4344-b1e1-c83dd77c925c" (UID: "8e76bd34-6d75-4344-b1e1-c83dd77c925c"). InnerVolumeSpecName "kube-api-access-g2s7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:20:47.352156 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.352122 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util" (OuterVolumeSpecName: "util") pod "8e76bd34-6d75-4344-b1e1-c83dd77c925c" (UID: "8e76bd34-6d75-4344-b1e1-c83dd77c925c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:20:47.448706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.448673 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2s7d\" (UniqueName: \"kubernetes.io/projected/8e76bd34-6d75-4344-b1e1-c83dd77c925c-kube-api-access-g2s7d\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:20:47.448706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.448703 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:20:47.448706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:47.448712 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e76bd34-6d75-4344-b1e1-c83dd77c925c-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:20:48.077670 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:48.077643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" Jul 23 18:20:48.077670 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:48.077654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tqx6r" event={"ID":"8e76bd34-6d75-4344-b1e1-c83dd77c925c","Type":"ContainerDied","Data":"d4592c3af6432589e3c8f0931b31f9e79049bf70f0e6458bd0c02e718d0aba77"} Jul 23 18:20:48.077897 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:48.077690 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4592c3af6432589e3c8f0931b31f9e79049bf70f0e6458bd0c02e718d0aba77" Jul 23 18:20:52.174463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:52.174426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:52.998282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:52.998250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:54.091188 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091155 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p"] Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091416 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="extract" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091428 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="extract" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091443 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="pull" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091448 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="pull" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091456 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="util" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091461 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="util" Jul 23 18:20:54.091594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.091509 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e76bd34-6d75-4344-b1e1-c83dd77c925c" containerName="extract" Jul 23 18:20:54.094851 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.094826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.097238 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.097201 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:20:54.097348 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.097245 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 18:20:54.097404 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.097352 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-z7tmk\"" Jul 23 18:20:54.107468 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.107449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p"] Jul 23 18:20:54.195667 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.195643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0a5982-38bc-479b-beab-6695fd0207ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.195763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.195695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jtj\" (UniqueName: \"kubernetes.io/projected/ba0a5982-38bc-479b-beab-6695fd0207ca-kube-api-access-d7jtj\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.296884 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.296853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jtj\" (UniqueName: \"kubernetes.io/projected/ba0a5982-38bc-479b-beab-6695fd0207ca-kube-api-access-d7jtj\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.296988 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.296901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0a5982-38bc-479b-beab-6695fd0207ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.297256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.297218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0a5982-38bc-479b-beab-6695fd0207ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.304784 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.304766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jtj\" (UniqueName: \"kubernetes.io/projected/ba0a5982-38bc-479b-beab-6695fd0207ca-kube-api-access-d7jtj\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmq8p\" (UID: \"ba0a5982-38bc-479b-beab-6695fd0207ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.403459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.403391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" Jul 23 18:20:54.535746 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:54.535707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p"] Jul 23 18:20:54.540761 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:20:54.540733 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0a5982_38bc_479b_beab_6695fd0207ca.slice/crio-ddd18caf0a6aeacb1a50e5cc002f451d0898851a0b1c57e77b40dc7164898981 WatchSource:0}: Error finding container ddd18caf0a6aeacb1a50e5cc002f451d0898851a0b1c57e77b40dc7164898981: Status 404 returned error can't find the container with id ddd18caf0a6aeacb1a50e5cc002f451d0898851a0b1c57e77b40dc7164898981 Jul 23 18:20:55.104862 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:55.104833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" event={"ID":"ba0a5982-38bc-479b-beab-6695fd0207ca","Type":"ContainerStarted","Data":"ddd18caf0a6aeacb1a50e5cc002f451d0898851a0b1c57e77b40dc7164898981"} Jul 23 18:20:55.789631 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:55.789591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:55.890337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:55.890284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:56.670432 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:56.670397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:56.776133 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:56.776098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:57.114372 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:57.114342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" event={"ID":"ba0a5982-38bc-479b-beab-6695fd0207ca","Type":"ContainerStarted","Data":"15e8e227be6390688f7150da1cbe23338f3ba9b1a3bfdfdc40dc726f35658bda"} Jul 23 18:20:57.135662 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:57.135615 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmq8p" podStartSLOduration=1.610332778 podStartE2EDuration="3.135598266s" totalImagesPullingTime="1.525265488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:20:57.133374894 +0000 UTC m=+283.864050001" watchObservedRunningTime="2026-07-23 18:20:57.135598266 +0000 UTC m=+283.866273358" Jul 23 18:20:57.469325 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:57.469293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:57.607584 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:57.607551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:58.201903 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.201872 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r"] Jul 23 18:20:58.204368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.204346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.207077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.207059 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:20:58.208085 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.208066 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5jb4d\"" Jul 23 18:20:58.208202 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.208083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:20:58.213287 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.213266 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r"] Jul 23 18:20:58.326396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.326363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgb5g\" (UniqueName: \"kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.326581 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.326408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.326581 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.326438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.427480 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.427452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.427621 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.427501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgb5g\" (UniqueName: \"kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.427621 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.427533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.427841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.427822 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.427900 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.427884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.438242 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.438203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgb5g\" (UniqueName: \"kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.513253 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.513174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:20:58.649800 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:58.649744 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r"] Jul 23 18:20:58.651835 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:20:58.651813 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e06e7b9_19fe_4392_a3e0_7774116e9693.slice/crio-af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1 WatchSource:0}: Error finding container af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1: Status 404 returned error can't find the container with id af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1 Jul 23 18:20:59.073787 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:59.073753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:20:59.123922 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:59.123889 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerID="4f66b8b33b01fe9c8a119ee4af68498df005176ddaa49dbae9b33de06a6e915a" exitCode=0 Jul 23 18:20:59.124066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:59.123976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" event={"ID":"5e06e7b9-19fe-4392-a3e0-7774116e9693","Type":"ContainerDied","Data":"4f66b8b33b01fe9c8a119ee4af68498df005176ddaa49dbae9b33de06a6e915a"} Jul 23 18:20:59.124066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:59.124021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" event={"ID":"5e06e7b9-19fe-4392-a3e0-7774116e9693","Type":"ContainerStarted","Data":"af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1"} Jul 23 18:20:59.273081 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:20:59.273053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:02.138559 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:02.138524 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerID="1ff0365ba7e6de96bb5cdbc4bcf936fc584a4aac1398582fe524c01c125f1227" exitCode=0 Jul 23 18:21:02.138945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:02.138611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" event={"ID":"5e06e7b9-19fe-4392-a3e0-7774116e9693","Type":"ContainerDied","Data":"1ff0365ba7e6de96bb5cdbc4bcf936fc584a4aac1398582fe524c01c125f1227"} Jul 23 18:21:02.780463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:02.780429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:02.837615 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:02.837588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:03.144401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:03.144321 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerID="6e58c39f1a81573f36e462f17856d0746a166622002de1e7dd3a5fda492200fe" exitCode=0 Jul 23 18:21:03.144401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:03.144371 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" event={"ID":"5e06e7b9-19fe-4392-a3e0-7774116e9693","Type":"ContainerDied","Data":"6e58c39f1a81573f36e462f17856d0746a166622002de1e7dd3a5fda492200fe"} Jul 23 18:21:04.269727 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.269706 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:21:04.373853 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.373825 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util\") pod \"5e06e7b9-19fe-4392-a3e0-7774116e9693\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " Jul 23 18:21:04.374031 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.373875 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgb5g\" (UniqueName: \"kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g\") pod \"5e06e7b9-19fe-4392-a3e0-7774116e9693\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " Jul 23 18:21:04.374031 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.373903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle\") pod \"5e06e7b9-19fe-4392-a3e0-7774116e9693\" (UID: \"5e06e7b9-19fe-4392-a3e0-7774116e9693\") " Jul 23 18:21:04.374296 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.374275 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle" (OuterVolumeSpecName: "bundle") pod "5e06e7b9-19fe-4392-a3e0-7774116e9693" (UID: "5e06e7b9-19fe-4392-a3e0-7774116e9693"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:21:04.375957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.375933 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g" (OuterVolumeSpecName: "kube-api-access-hgb5g") pod "5e06e7b9-19fe-4392-a3e0-7774116e9693" (UID: "5e06e7b9-19fe-4392-a3e0-7774116e9693"). InnerVolumeSpecName "kube-api-access-hgb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:21:04.378811 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.378781 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util" (OuterVolumeSpecName: "util") pod "5e06e7b9-19fe-4392-a3e0-7774116e9693" (UID: "5e06e7b9-19fe-4392-a3e0-7774116e9693"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:21:04.474553 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.474531 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:04.474651 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.474553 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgb5g\" (UniqueName: \"kubernetes.io/projected/5e06e7b9-19fe-4392-a3e0-7774116e9693-kube-api-access-hgb5g\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:04.474651 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:04.474570 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e06e7b9-19fe-4392-a3e0-7774116e9693-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:05.154623 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:05.154590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" event={"ID":"5e06e7b9-19fe-4392-a3e0-7774116e9693","Type":"ContainerDied","Data":"af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1"} Jul 23 18:21:05.154785 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:05.154638 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af98bc3ae38792ea02c667fbf4e09734ab2fbc2218895ac37af4c922004727e1" Jul 23 18:21:05.154785 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:05.154600 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs265r" Jul 23 18:21:06.287700 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.287671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:06.964769 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.964733 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kc82s"] Jul 23 18:21:06.965003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.964991 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="pull" Jul 23 18:21:06.965053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965005 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="pull" Jul 23 18:21:06.965053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965014 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="extract" Jul 23 18:21:06.965053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965019 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="extract" Jul 23 18:21:06.965053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965030 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="util" Jul 23 18:21:06.965053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965036 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="util" Jul 23 18:21:06.965250 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.965084 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e06e7b9-19fe-4392-a3e0-7774116e9693" containerName="extract" Jul 23 18:21:06.967511 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.967496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:06.969895 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.969872 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 18:21:06.970102 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.970083 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4n9ln\"" Jul 23 18:21:06.970765 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.970752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 18:21:06.977306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.977284 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kc82s"] Jul 23 18:21:06.992280 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.992244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8nc\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-kube-api-access-7z8nc\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:06.992424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:06.992314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-bound-sa-token\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.093278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.093217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8nc\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-kube-api-access-7z8nc\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.093461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.093311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-bound-sa-token\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.101459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.101428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-bound-sa-token\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.101569 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.101512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8nc\" (UniqueName: \"kubernetes.io/projected/587435e6-c4ab-4a04-91cf-cffc36d3fd48-kube-api-access-7z8nc\") pod \"cert-manager-55b4d954d5-kc82s\" (UID: \"587435e6-c4ab-4a04-91cf-cffc36d3fd48\") " pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.276758 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.276693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kc82s" Jul 23 18:21:07.400407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:07.400378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kc82s"] Jul 23 18:21:07.402382 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:21:07.402355 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod587435e6_c4ab_4a04_91cf_cffc36d3fd48.slice/crio-b27015679bedc222cbaed5ab25e26b066817ec8134e9023782439c098206e684 WatchSource:0}: Error finding container b27015679bedc222cbaed5ab25e26b066817ec8134e9023782439c098206e684: Status 404 returned error can't find the container with id b27015679bedc222cbaed5ab25e26b066817ec8134e9023782439c098206e684 Jul 23 18:21:08.173059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:08.173019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kc82s" event={"ID":"587435e6-c4ab-4a04-91cf-cffc36d3fd48","Type":"ContainerStarted","Data":"b27015679bedc222cbaed5ab25e26b066817ec8134e9023782439c098206e684"} Jul 23 18:21:11.189572 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:11.189538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kc82s" event={"ID":"587435e6-c4ab-4a04-91cf-cffc36d3fd48","Type":"ContainerStarted","Data":"e643ad328b0e0b3ba1db275437f5318eb7fe3933aac5475d6bab682e8f0cc77c"} Jul 23 18:21:11.204913 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:11.204866 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kc82s" podStartSLOduration=2.476561658 podStartE2EDuration="5.204851169s" totalImagesPullingTime="2.728289511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:11.204333188 +0000 UTC m=+297.935008304" watchObservedRunningTime="2026-07-23 18:21:11.204851169 +0000 UTC m=+297.935526259" Jul 23 18:21:13.396677 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.396639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:13.494220 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.494186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:13.685723 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.685664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:21:13.686292 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.686272 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:21:13.688990 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.688970 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:21:13.689853 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.689835 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:21:13.695788 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:13.695766 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 18:21:19.829908 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:19.829873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:21.186475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:21.186436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:21.675147 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:21.675113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:22.275605 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.275565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:22.408279 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.408250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:22.855248 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.855153 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44"] Jul 23 18:21:22.858140 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.858112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.860664 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.860640 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:21:22.860764 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.860640 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:21:22.861565 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.861540 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5jb4d\"" Jul 23 18:21:22.866539 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.866505 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44"] Jul 23 18:21:22.892474 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.892449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtpj\" (UniqueName: \"kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.892578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.892495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.892578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.892570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.993704 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.993675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtpj\" (UniqueName: \"kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.993814 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.993709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.993814 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.993738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.994089 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.994072 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:22.994130 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:22.994105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:23.001681 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:23.001652 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtpj\" (UniqueName: \"kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:23.169186 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:23.169131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:23.296714 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:23.296692 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44"] Jul 23 18:21:23.298720 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:21:23.298696 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e66c4e_96b3_4764_9ef7_f0f16d76d3f8.slice/crio-86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66 WatchSource:0}: Error finding container 86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66: Status 404 returned error can't find the container with id 86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66 Jul 23 18:21:24.245535 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:24.245496 2576 generic.go:356] "Generic (PLEG): container finished" podID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerID="2a2eccf14bd73dce9523d8f2fe57c92de3e475378d06c8c6aa088ec8a172ba26" exitCode=0 Jul 23 18:21:24.245763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:24.245588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" event={"ID":"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8","Type":"ContainerDied","Data":"2a2eccf14bd73dce9523d8f2fe57c92de3e475378d06c8c6aa088ec8a172ba26"} Jul 23 18:21:24.245763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:24.245655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" event={"ID":"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8","Type":"ContainerStarted","Data":"86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66"} Jul 23 18:21:24.246043 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:24.246023 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:21:25.251514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:25.251476 2576 generic.go:356] "Generic (PLEG): container finished" podID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerID="c4e7c4ae6d0e6417155357910fce0ca2303f9980b5028f945eaf8b32d807ceaa" exitCode=0 Jul 23 18:21:25.251908 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:25.251553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" event={"ID":"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8","Type":"ContainerDied","Data":"c4e7c4ae6d0e6417155357910fce0ca2303f9980b5028f945eaf8b32d807ceaa"} Jul 23 18:21:26.257504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:26.257470 2576 generic.go:356] "Generic (PLEG): container finished" podID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerID="fd7ddf60a87e85a7c24a120789373d95df91e3a8327797b7965875e393039867" exitCode=0 Jul 23 18:21:26.257907 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:26.257562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" event={"ID":"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8","Type":"ContainerDied","Data":"fd7ddf60a87e85a7c24a120789373d95df91e3a8327797b7965875e393039867"} Jul 23 18:21:27.381683 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.381656 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:27.426951 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.426920 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjtpj\" (UniqueName: \"kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj\") pod \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " Jul 23 18:21:27.427104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.426960 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util\") pod \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " Jul 23 18:21:27.427104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.426998 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle\") pod \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\" (UID: \"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8\") " Jul 23 18:21:27.427771 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.427743 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle" (OuterVolumeSpecName: "bundle") pod "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" (UID: "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:21:27.428967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.428943 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj" (OuterVolumeSpecName: "kube-api-access-jjtpj") pod "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" (UID: "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8"). InnerVolumeSpecName "kube-api-access-jjtpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:21:27.432072 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.432034 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util" (OuterVolumeSpecName: "util") pod "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" (UID: "28e66c4e-96b3-4764-9ef7-f0f16d76d3f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:21:27.527578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.527499 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:27.527578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.527529 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtpj\" (UniqueName: \"kubernetes.io/projected/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-kube-api-access-jjtpj\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:27.527578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:27.527540 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28e66c4e-96b3-4764-9ef7-f0f16d76d3f8-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:21:28.270016 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:28.269974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" event={"ID":"28e66c4e-96b3-4764-9ef7-f0f16d76d3f8","Type":"ContainerDied","Data":"86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66"} Jul 23 18:21:28.270016 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:28.270011 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afx9g44" Jul 23 18:21:28.270302 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:28.270037 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86165776d4e45c4b6223182460d8b390286c01c4c79ebb4fc576ebf4998fce66" Jul 23 18:21:34.390832 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:34.390795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:34.494809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:34.494779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.007854 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.007789 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.241897 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.241854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.347694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.347612 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.367412 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367383 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v"] Jul 23 18:21:35.367647 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367635 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="util" Jul 23 18:21:35.367694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367649 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="util" Jul 23 18:21:35.367694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="extract" Jul 23 18:21:35.367694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367662 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="extract" Jul 23 18:21:35.367694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367669 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="pull" Jul 23 18:21:35.367694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367675 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="pull" Jul 23 18:21:35.367839 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.367725 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e66c4e-96b3-4764-9ef7-f0f16d76d3f8" containerName="extract" Jul 23 18:21:35.370422 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.370407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.373097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.373074 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 18:21:35.373097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.373089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-565dp\"" Jul 23 18:21:35.373312 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.373081 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 18:21:35.373312 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.373113 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 18:21:35.373312 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.373120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 18:21:35.382367 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.382347 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v"] Jul 23 18:21:35.384716 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.384694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.384803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.384739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.384803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.384770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.384880 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.384833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8nb\" (UniqueName: \"kubernetes.io/projected/3d0661c5-019a-48a6-9209-cd00bb3192bb-kube-api-access-4m8nb\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.384914 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.384886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486131 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8nb\" (UniqueName: \"kubernetes.io/projected/3d0661c5-019a-48a6-9209-cd00bb3192bb-kube-api-access-4m8nb\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.486540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.486501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-manifests\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.488411 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.488393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d0661c5-019a-48a6-9209-cd00bb3192bb-tmp\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.488717 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.488700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.488761 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.488741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d0661c5-019a-48a6-9209-cd00bb3192bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.507793 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.507767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8nb\" (UniqueName: \"kubernetes.io/projected/3d0661c5-019a-48a6-9209-cd00bb3192bb-kube-api-access-4m8nb\") pod \"opendatahub-operator-controller-manager-77bdbd87db-fjc9v\" (UID: \"3d0661c5-019a-48a6-9209-cd00bb3192bb\") " pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.573534 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.573511 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.674796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.674736 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.680526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.680498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:35.775815 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.775768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.809850 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.809829 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v"] Jul 23 18:21:35.812337 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:21:35.812310 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0661c5_019a_48a6_9209_cd00bb3192bb.slice/crio-2771fa548cb2b81f8c6b981f2fe52fbe522619710cf6276088321d1b1b3b72a8 WatchSource:0}: Error finding container 2771fa548cb2b81f8c6b981f2fe52fbe522619710cf6276088321d1b1b3b72a8: Status 404 returned error can't find the container with id 2771fa548cb2b81f8c6b981f2fe52fbe522619710cf6276088321d1b1b3b72a8 Jul 23 18:21:35.848713 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.848682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:35.919096 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:35.919067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:36.026904 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:36.026876 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:36.126889 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:36.126859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:36.311990 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:36.311907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" event={"ID":"3d0661c5-019a-48a6-9209-cd00bb3192bb","Type":"ContainerStarted","Data":"2771fa548cb2b81f8c6b981f2fe52fbe522619710cf6276088321d1b1b3b72a8"} Jul 23 18:21:39.327220 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:39.327151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" event={"ID":"3d0661c5-019a-48a6-9209-cd00bb3192bb","Type":"ContainerStarted","Data":"bd15e19a80b7488690545ca45c8b0aef4527bc34c923e923d95c3bdbda9b4d2d"} Jul 23 18:21:39.963319 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:39.963287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:40.057354 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:40.057320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:40.333671 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:40.333597 2576 generic.go:356] "Generic (PLEG): container finished" podID="3d0661c5-019a-48a6-9209-cd00bb3192bb" containerID="bd15e19a80b7488690545ca45c8b0aef4527bc34c923e923d95c3bdbda9b4d2d" exitCode=0 Jul 23 18:21:40.333671 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:40.333661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" event={"ID":"3d0661c5-019a-48a6-9209-cd00bb3192bb","Type":"ContainerDied","Data":"bd15e19a80b7488690545ca45c8b0aef4527bc34c923e923d95c3bdbda9b4d2d"} Jul 23 18:21:41.340339 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:41.340292 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" event={"ID":"3d0661c5-019a-48a6-9209-cd00bb3192bb","Type":"ContainerStarted","Data":"ba5865139d14b12a677e3271185db39304b226cad70388b70b95a56cb2d7e683"} Jul 23 18:21:41.340936 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:41.340389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:41.367083 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:41.367040 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" podStartSLOduration=1.748785683 podStartE2EDuration="6.367021761s" totalImagesPullingTime="3.579331882s" totalInitContainerRuntime="1.038904196s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:41.366052531 +0000 UTC m=+328.096727622" watchObservedRunningTime="2026-07-23 18:21:41.367021761 +0000 UTC m=+328.097696852" Jul 23 18:21:46.544257 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:46.544185 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:46.645847 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:46.645787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:52.353092 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:52.353058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77bdbd87db-fjc9v" Jul 23 18:21:54.400396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.400354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:54.566067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.566035 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:54.621967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.621940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r"] Jul 23 18:21:54.634157 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.634131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.639056 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.639033 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 18:21:54.639697 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.639671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r"] Jul 23 18:21:54.639951 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.639932 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 18:21:54.640058 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.639971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:21:54.640058 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.640007 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6gqr7\"" Jul 23 18:21:54.640058 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.640022 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 18:21:54.640329 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.640304 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 18:21:54.736464 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.736441 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.736575 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.736473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.736575 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.736502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-manager-config\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.736575 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.736530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8vn\" (UniqueName: \"kubernetes.io/projected/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-kube-api-access-pd8vn\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.837003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.836981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.837123 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.837008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.837123 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.837036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-manager-config\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.837123 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.837072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8vn\" (UniqueName: \"kubernetes.io/projected/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-kube-api-access-pd8vn\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.837763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.837744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-manager-config\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.839514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.839491 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.839580 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.839531 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-metrics-cert\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.852190 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.852166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8vn\" (UniqueName: \"kubernetes.io/projected/8d315cd6-9cc2-4df2-864d-bcd1f7667b2d-kube-api-access-pd8vn\") pod \"lws-controller-manager-778f8cb4df-gcq9r\" (UID: \"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d\") " pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:54.870210 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.870183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:54.944347 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:54.944324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:55.083413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.083387 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r"] Jul 23 18:21:55.084911 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:21:55.084884 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d315cd6_9cc2_4df2_864d_bcd1f7667b2d.slice/crio-82b2b473e5c77b38722c0769c0a230a4c9a0845ef8999a9b410c539cd3c483a2 WatchSource:0}: Error finding container 82b2b473e5c77b38722c0769c0a230a4c9a0845ef8999a9b410c539cd3c483a2: Status 404 returned error can't find the container with id 82b2b473e5c77b38722c0769c0a230a4c9a0845ef8999a9b410c539cd3c483a2 Jul 23 18:21:55.357802 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.357742 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:55.409629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.409601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" event={"ID":"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d","Type":"ContainerStarted","Data":"82b2b473e5c77b38722c0769c0a230a4c9a0845ef8999a9b410c539cd3c483a2"} Jul 23 18:21:55.454726 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.454699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:55.519573 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.519542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:55.675474 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:55.675392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:56.308797 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.308768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:56.392081 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.392058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:56.457067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.457036 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2blvr"] Jul 23 18:21:56.460180 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.460164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.462524 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.462503 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pmrzp\"" Jul 23 18:21:56.462623 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.462548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 18:21:56.474916 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.474890 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2blvr"] Jul 23 18:21:56.549482 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.549460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmj7n\" (UniqueName: \"kubernetes.io/projected/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-kube-api-access-cmj7n\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.549574 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.549502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-tls-certs\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.650025 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.649953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-tls-certs\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.650025 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.650017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmj7n\" (UniqueName: \"kubernetes.io/projected/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-kube-api-access-cmj7n\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.652489 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.652463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-tls-certs\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.673474 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.673447 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmj7n\" (UniqueName: \"kubernetes.io/projected/ec432d0f-6b9a-45d9-95f2-9f43e82104fe-kube-api-access-cmj7n\") pod \"model-serving-api-54d89bd5fc-2blvr\" (UID: \"ec432d0f-6b9a-45d9-95f2-9f43e82104fe\") " pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.765254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.765214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:56.770009 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.769989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:56.912486 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:56.912465 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2blvr"] Jul 23 18:21:56.914473 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:21:56.914448 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec432d0f_6b9a_45d9_95f2_9f43e82104fe.slice/crio-fd9974c18d585e1a65e1b586ac6be646436934540a4e4af766a8bb8796949094 WatchSource:0}: Error finding container fd9974c18d585e1a65e1b586ac6be646436934540a4e4af766a8bb8796949094: Status 404 returned error can't find the container with id fd9974c18d585e1a65e1b586ac6be646436934540a4e4af766a8bb8796949094 Jul 23 18:21:57.419235 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:57.419185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" event={"ID":"ec432d0f-6b9a-45d9-95f2-9f43e82104fe","Type":"ContainerStarted","Data":"fd9974c18d585e1a65e1b586ac6be646436934540a4e4af766a8bb8796949094"} Jul 23 18:21:57.420890 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:57.420863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" event={"ID":"8d315cd6-9cc2-4df2-864d-bcd1f7667b2d","Type":"ContainerStarted","Data":"6fc3d69e4315e23422db3c7f2b25f9e917e746338f9ef7751d496ced4a13d0f1"} Jul 23 18:21:57.421024 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:57.420987 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:21:57.439204 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:57.439160 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" podStartSLOduration=2.164919614 podStartE2EDuration="3.439144s" totalImagesPullingTime="1.274224386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:57.438251478 +0000 UTC m=+344.168926569" watchObservedRunningTime="2026-07-23 18:21:57.439144 +0000 UTC m=+344.169819090" Jul 23 18:21:57.898340 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:57.898289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:58.648915 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:58.648859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:58.748504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:58.748456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:21:59.435281 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:59.435184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" event={"ID":"ec432d0f-6b9a-45d9-95f2-9f43e82104fe","Type":"ContainerStarted","Data":"e03bbc6ae854c09c966faf274259e8431919369d9f797afd7dda8a33d97bd711"} Jul 23 18:21:59.435640 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:59.435334 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:21:59.451641 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:21:59.451597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" podStartSLOduration=1.40975355 podStartE2EDuration="3.451583591s" totalImagesPullingTime="2.041830041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:21:59.450563146 +0000 UTC m=+346.181238258" watchObservedRunningTime="2026-07-23 18:21:59.451583591 +0000 UTC m=+346.182258719" Jul 23 18:22:02.108708 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.108672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:02.209322 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.209285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:02.918363 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.918330 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm"] Jul 23 18:22:02.922938 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.922915 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.925515 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.925494 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 18:22:02.925640 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.925518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fvmmt\"" Jul 23 18:22:02.936374 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.936351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm"] Jul 23 18:22:02.959018 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.958998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:02.995304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-kube-api-access-thm96\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995391 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995391 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995391 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:02.995616 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:02.995538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096165 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-kube-api-access-thm96\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096448 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096448 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096448 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096599 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096655 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096716 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096771 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.096888 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.096867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.097206 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.097180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.098508 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.098479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.098777 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.098760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.105254 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.105209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.105337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.105302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/87d25df1-bb18-4b7c-a2eb-3ed44efe72bf-kube-api-access-thm96\") pod \"data-science-gateway-data-science-gateway-class-6576859cf42hzhm\" (UID: \"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.161291 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.161259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:03.232788 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.232762 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:03.262831 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.262803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:03.398753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.398723 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm"] Jul 23 18:22:03.400007 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:22:03.399979 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d25df1_bb18_4b7c_a2eb_3ed44efe72bf.slice/crio-909c5374bbd7a8b2148a8d1813d4e74a864bf8e2dcb92fbe2a922ef344d64cf5 WatchSource:0}: Error finding container 909c5374bbd7a8b2148a8d1813d4e74a864bf8e2dcb92fbe2a922ef344d64cf5: Status 404 returned error can't find the container with id 909c5374bbd7a8b2148a8d1813d4e74a864bf8e2dcb92fbe2a922ef344d64cf5 Jul 23 18:22:03.455442 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:03.455416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" event={"ID":"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf","Type":"ContainerStarted","Data":"909c5374bbd7a8b2148a8d1813d4e74a864bf8e2dcb92fbe2a922ef344d64cf5"} Jul 23 18:22:06.199485 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.199396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:06.215603 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.215558 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:22:06.215709 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.215667 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:22:06.215753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.215717 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:22:06.270419 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.270396 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk"] Jul 23 18:22:06.273779 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.273759 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.276963 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.276916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 18:22:06.277087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.277004 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-p8b4f\"" Jul 23 18:22:06.285339 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.285318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk"] Jul 23 18:22:06.291003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.290970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:06.391752 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.391719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:06.424702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.424670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243cdeaf-6f1d-4b7d-9512-8962daf175a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.424702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.424710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fv6q\" (UniqueName: \"kubernetes.io/projected/243cdeaf-6f1d-4b7d-9512-8962daf175a4-kube-api-access-2fv6q\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.471887 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.471785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" event={"ID":"87d25df1-bb18-4b7c-a2eb-3ed44efe72bf","Type":"ContainerStarted","Data":"500b2c54dba2c4e3afcea52cc38068f61f2ec2746d61749949fb94f24298e181"} Jul 23 18:22:06.501246 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.499252 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" podStartSLOduration=1.685951517 podStartE2EDuration="4.499208928s" totalImagesPullingTime="2.813257411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:22:06.496202222 +0000 UTC m=+353.226877313" watchObservedRunningTime="2026-07-23 18:22:06.499208928 +0000 UTC m=+353.229884020" Jul 23 18:22:06.508944 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.508891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:06.525627 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.525588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243cdeaf-6f1d-4b7d-9512-8962daf175a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.525627 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.525626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fv6q\" (UniqueName: \"kubernetes.io/projected/243cdeaf-6f1d-4b7d-9512-8962daf175a4-kube-api-access-2fv6q\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.528240 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.528194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/243cdeaf-6f1d-4b7d-9512-8962daf175a4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.543154 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.543124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fv6q\" (UniqueName: \"kubernetes.io/projected/243cdeaf-6f1d-4b7d-9512-8962daf175a4-kube-api-access-2fv6q\") pod \"llmisvc-controller-manager-777c6bd7d5-z6fnk\" (UID: \"243cdeaf-6f1d-4b7d-9512-8962daf175a4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.585519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.585486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:06.653539 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.653503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:06.726202 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:06.726174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk"] Jul 23 18:22:06.728454 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:22:06.728425 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod243cdeaf_6f1d_4b7d_9512_8962daf175a4.slice/crio-a2de39e0aee4a36310d7676d4a3342a5cf44d56b94db542fc8d57f524529f7f1 WatchSource:0}: Error finding container a2de39e0aee4a36310d7676d4a3342a5cf44d56b94db542fc8d57f524529f7f1: Status 404 returned error can't find the container with id a2de39e0aee4a36310d7676d4a3342a5cf44d56b94db542fc8d57f524529f7f1 Jul 23 18:22:07.232939 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:07.232907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:07.238077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:07.238054 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:07.479419 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:07.479358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" event={"ID":"243cdeaf-6f1d-4b7d-9512-8962daf175a4","Type":"ContainerStarted","Data":"a2de39e0aee4a36310d7676d4a3342a5cf44d56b94db542fc8d57f524529f7f1"} Jul 23 18:22:07.480037 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:07.479997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:07.480966 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:07.480943 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf42hzhm" Jul 23 18:22:08.427763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:08.427729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778f8cb4df-gcq9r" Jul 23 18:22:10.444333 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:10.444309 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2blvr" Jul 23 18:22:11.500724 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:11.500685 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" event={"ID":"243cdeaf-6f1d-4b7d-9512-8962daf175a4","Type":"ContainerStarted","Data":"bf306243e9958e06470496da582915c09aae52226315b406d0135c83822f18c8"} Jul 23 18:22:11.501115 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:11.500785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:11.516274 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:11.516215 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" podStartSLOduration=1.768883415 podStartE2EDuration="5.516201756s" totalImagesPullingTime="3.747318341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:22:11.514995542 +0000 UTC m=+358.245670635" watchObservedRunningTime="2026-07-23 18:22:11.516201756 +0000 UTC m=+358.246876846" Jul 23 18:22:23.837382 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:23.837350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:42.507871 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:42.507832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z6fnk" Jul 23 18:22:55.960007 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:55.959959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:56.058403 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:56.058360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:56.157509 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:56.157477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:56.893883 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:56.893834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:22:57.104248 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:22:57.104205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:11.706769 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.706733 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x"] Jul 23 18:23:11.710279 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.710260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.712844 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.712812 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w6tlp\"" Jul 23 18:23:11.712999 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.712891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 18:23:11.712999 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.712988 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 18:23:11.718420 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.718402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x"] Jul 23 18:23:11.819611 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.819585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.819738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.819622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx7c\" (UniqueName: \"kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.819738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.819654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.920905 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.920878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.921026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.920913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx7c\" (UniqueName: \"kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.921026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.920964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.921386 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.921364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.921386 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.921377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:11.928872 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:11.928836 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx7c\" (UniqueName: \"kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:12.021079 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.021003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:12.107794 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.107765 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2"] Jul 23 18:23:12.111401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.111381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.128379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.128357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2"] Jul 23 18:23:12.166124 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.165495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x"] Jul 23 18:23:12.223982 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.223958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.224084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.223998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.224084 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.224030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdwf9\" (UniqueName: \"kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.325340 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.325282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.325340 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.325322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.325473 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.325344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdwf9\" (UniqueName: \"kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.325651 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.325628 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.325705 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.325686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.333133 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.333103 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdwf9\" (UniqueName: \"kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.422408 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.422385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:12.523757 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.523732 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv"] Jul 23 18:23:12.527461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.527441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.536807 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.536779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv"] Jul 23 18:23:12.553889 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.553871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2"] Jul 23 18:23:12.555951 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:23:12.555914 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793 WatchSource:0}: Error finding container 10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793: Status 404 returned error can't find the container with id 10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793 Jul 23 18:23:12.631043 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.631015 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.631162 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.631066 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.631162 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.631088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrnx\" (UniqueName: \"kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.731840 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.731817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.732119 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.731865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.732119 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.731959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrnx\" (UniqueName: \"kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.732193 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.732134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.732243 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.732198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.744723 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.744701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrnx\" (UniqueName: \"kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.798218 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.798194 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec25ebc5-75f5-4022-a96e-58091705762e" containerID="fb03739b549fca1dddeb0fc6d7f245f603a250ae98206edcbadd7d19a2092b06" exitCode=0 Jul 23 18:23:12.798332 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.798284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" event={"ID":"ec25ebc5-75f5-4022-a96e-58091705762e","Type":"ContainerDied","Data":"fb03739b549fca1dddeb0fc6d7f245f603a250ae98206edcbadd7d19a2092b06"} Jul 23 18:23:12.798376 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.798335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" event={"ID":"ec25ebc5-75f5-4022-a96e-58091705762e","Type":"ContainerStarted","Data":"7140746d773d4c171b8d05a5e8fdf9c6aacf3d039b9239aac6ddf400540275a9"} Jul 23 18:23:12.799934 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.799914 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerID="eb60f55ae9601625bec3bada6d1b34c3c9ff943acb5d784ca2b619b42f576a05" exitCode=0 Jul 23 18:23:12.800019 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.799978 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" event={"ID":"0d63f438-3869-4ae6-bed3-e77a7df90a3e","Type":"ContainerDied","Data":"eb60f55ae9601625bec3bada6d1b34c3c9ff943acb5d784ca2b619b42f576a05"} Jul 23 18:23:12.800067 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.800023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" event={"ID":"0d63f438-3869-4ae6-bed3-e77a7df90a3e","Type":"ContainerStarted","Data":"10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793"} Jul 23 18:23:12.839104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.839056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:12.904644 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.904616 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd"] Jul 23 18:23:12.908335 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.908315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:12.917217 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.917130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd"] Jul 23 18:23:12.967455 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:12.967436 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv"] Jul 23 18:23:12.969339 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:23:12.969316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94fe07c_a82e_4232_90ce_b3fd4731b3b2.slice/crio-ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c WatchSource:0}: Error finding container ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c: Status 404 returned error can't find the container with id ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c Jul 23 18:23:13.034062 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.034040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.034160 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.034080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.034160 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.034137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzfh\" (UniqueName: \"kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.135501 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.135432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzfh\" (UniqueName: \"kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.135501 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.135491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.135630 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.135524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.135823 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.135809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.135881 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.135865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.143528 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.143508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzfh\" (UniqueName: \"kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.220373 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.220354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:13.347381 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.347354 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd"] Jul 23 18:23:13.349347 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:23:13.349320 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7b28d3_ef3d_448c_ab03_16c7c68ba088.slice/crio-a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a WatchSource:0}: Error finding container a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a: Status 404 returned error can't find the container with id a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a Jul 23 18:23:13.806268 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.806244 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec25ebc5-75f5-4022-a96e-58091705762e" containerID="2374637f3c1e531f0eae5f2321548d2d7dd86b626127cc3902e3832ca6ec5103" exitCode=0 Jul 23 18:23:13.806602 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.806328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" event={"ID":"ec25ebc5-75f5-4022-a96e-58091705762e","Type":"ContainerDied","Data":"2374637f3c1e531f0eae5f2321548d2d7dd86b626127cc3902e3832ca6ec5103"} Jul 23 18:23:13.808181 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.808163 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerID="cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639" exitCode=0 Jul 23 18:23:13.808276 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.808250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" event={"ID":"0d63f438-3869-4ae6-bed3-e77a7df90a3e","Type":"ContainerDied","Data":"cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639"} Jul 23 18:23:13.809738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.809709 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerID="677440c5fe1659bb9823ed259763e4ad47ca5e82ac62c8c81c6665af32b18278" exitCode=0 Jul 23 18:23:13.809841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.809810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" event={"ID":"4a7b28d3-ef3d-448c-ab03-16c7c68ba088","Type":"ContainerDied","Data":"677440c5fe1659bb9823ed259763e4ad47ca5e82ac62c8c81c6665af32b18278"} Jul 23 18:23:13.809903 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.809853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" event={"ID":"4a7b28d3-ef3d-448c-ab03-16c7c68ba088","Type":"ContainerStarted","Data":"a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a"} Jul 23 18:23:13.811380 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.811359 2576 generic.go:356] "Generic (PLEG): container finished" podID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerID="6d12ff8a5a15b5582d41d71a58ae812a6aa03777106cd2d54926609d7ee8d1f6" exitCode=0 Jul 23 18:23:13.811473 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.811435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" event={"ID":"e94fe07c-a82e-4232-90ce-b3fd4731b3b2","Type":"ContainerDied","Data":"6d12ff8a5a15b5582d41d71a58ae812a6aa03777106cd2d54926609d7ee8d1f6"} Jul 23 18:23:13.811473 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:13.811469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" event={"ID":"e94fe07c-a82e-4232-90ce-b3fd4731b3b2","Type":"ContainerStarted","Data":"ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c"} Jul 23 18:23:14.148538 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:14.148504 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:14.790201 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:14.790172 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:14.818626 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.818594 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerID="517cfe3c717294d7fa67541089bd0d932cc7b161aadcad902f1a8dca4e4b24b0" exitCode=0 Jul 23 18:23:14.818955 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.818684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" event={"ID":"0d63f438-3869-4ae6-bed3-e77a7df90a3e","Type":"ContainerDied","Data":"517cfe3c717294d7fa67541089bd0d932cc7b161aadcad902f1a8dca4e4b24b0"} Jul 23 18:23:14.820341 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.820320 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerID="b99304c627a0f82d16b655ee29aee9bfde0c85627a50d8594a87b61a3029d6b8" exitCode=0 Jul 23 18:23:14.820443 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.820345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" event={"ID":"4a7b28d3-ef3d-448c-ab03-16c7c68ba088","Type":"ContainerDied","Data":"b99304c627a0f82d16b655ee29aee9bfde0c85627a50d8594a87b61a3029d6b8"} Jul 23 18:23:14.822260 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.822214 2576 generic.go:356] "Generic (PLEG): container finished" podID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerID="432a61c438c2c23f9219905fa6689b3d20880099210a8cc32c170da3f1262074" exitCode=0 Jul 23 18:23:14.822372 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.822298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" event={"ID":"e94fe07c-a82e-4232-90ce-b3fd4731b3b2","Type":"ContainerDied","Data":"432a61c438c2c23f9219905fa6689b3d20880099210a8cc32c170da3f1262074"} Jul 23 18:23:14.824384 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.824365 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec25ebc5-75f5-4022-a96e-58091705762e" containerID="cd525545a954d4a61af472377b3a373d80e49e1df9fb80a6305aa9e632f15307" exitCode=0 Jul 23 18:23:14.824465 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:14.824390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" event={"ID":"ec25ebc5-75f5-4022-a96e-58091705762e","Type":"ContainerDied","Data":"cd525545a954d4a61af472377b3a373d80e49e1df9fb80a6305aa9e632f15307"} Jul 23 18:23:15.832620 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:15.832583 2576 generic.go:356] "Generic (PLEG): container finished" podID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerID="42b4dfdcf869129c36dbd74d9953c086233a8f8a4efaa5f35ec5d72fe78afc0d" exitCode=0 Jul 23 18:23:15.835104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:15.835076 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerID="5d190bf3c4a6b772c48684dda410d48912f58af9cbeb944fc9fc0b35995de3b0" exitCode=0 Jul 23 18:23:15.836052 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:15.835842 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" event={"ID":"e94fe07c-a82e-4232-90ce-b3fd4731b3b2","Type":"ContainerDied","Data":"42b4dfdcf869129c36dbd74d9953c086233a8f8a4efaa5f35ec5d72fe78afc0d"} Jul 23 18:23:15.836052 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:15.835895 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" event={"ID":"4a7b28d3-ef3d-448c-ab03-16c7c68ba088","Type":"ContainerDied","Data":"5d190bf3c4a6b772c48684dda410d48912f58af9cbeb944fc9fc0b35995de3b0"} Jul 23 18:23:15.971079 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:15.971050 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:16.003325 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.003299 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:16.056942 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.056919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util\") pod \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " Jul 23 18:23:16.057048 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.056954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdwf9\" (UniqueName: \"kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9\") pod \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " Jul 23 18:23:16.057048 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.056995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgx7c\" (UniqueName: \"kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c\") pod \"ec25ebc5-75f5-4022-a96e-58091705762e\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " Jul 23 18:23:16.057151 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.057053 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle\") pod \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\" (UID: \"0d63f438-3869-4ae6-bed3-e77a7df90a3e\") " Jul 23 18:23:16.057151 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.057108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle\") pod \"ec25ebc5-75f5-4022-a96e-58091705762e\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " Jul 23 18:23:16.057151 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.057141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util\") pod \"ec25ebc5-75f5-4022-a96e-58091705762e\" (UID: \"ec25ebc5-75f5-4022-a96e-58091705762e\") " Jul 23 18:23:16.057661 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.057630 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle" (OuterVolumeSpecName: "bundle") pod "0d63f438-3869-4ae6-bed3-e77a7df90a3e" (UID: "0d63f438-3869-4ae6-bed3-e77a7df90a3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:16.058121 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.058094 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle" (OuterVolumeSpecName: "bundle") pod "ec25ebc5-75f5-4022-a96e-58091705762e" (UID: "ec25ebc5-75f5-4022-a96e-58091705762e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:16.059365 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.059342 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c" (OuterVolumeSpecName: "kube-api-access-xgx7c") pod "ec25ebc5-75f5-4022-a96e-58091705762e" (UID: "ec25ebc5-75f5-4022-a96e-58091705762e"). InnerVolumeSpecName "kube-api-access-xgx7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:23:16.059365 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.059349 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9" (OuterVolumeSpecName: "kube-api-access-fdwf9") pod "0d63f438-3869-4ae6-bed3-e77a7df90a3e" (UID: "0d63f438-3869-4ae6-bed3-e77a7df90a3e"). InnerVolumeSpecName "kube-api-access-fdwf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:23:16.065988 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.065964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util" (OuterVolumeSpecName: "util") pod "0d63f438-3869-4ae6-bed3-e77a7df90a3e" (UID: "0d63f438-3869-4ae6-bed3-e77a7df90a3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:16.066109 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.066091 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util" (OuterVolumeSpecName: "util") pod "ec25ebc5-75f5-4022-a96e-58091705762e" (UID: "ec25ebc5-75f5-4022-a96e-58091705762e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.157989 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.158022 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec25ebc5-75f5-4022-a96e-58091705762e-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.158030 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.158038 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdwf9\" (UniqueName: \"kubernetes.io/projected/0d63f438-3869-4ae6-bed3-e77a7df90a3e-kube-api-access-fdwf9\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.158048 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgx7c\" (UniqueName: \"kubernetes.io/projected/ec25ebc5-75f5-4022-a96e-58091705762e-kube-api-access-xgx7c\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.158051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.158057 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d63f438-3869-4ae6-bed3-e77a7df90a3e-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:16.842258 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.842216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" Jul 23 18:23:16.842258 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.842244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x" event={"ID":"ec25ebc5-75f5-4022-a96e-58091705762e","Type":"ContainerDied","Data":"7140746d773d4c171b8d05a5e8fdf9c6aacf3d039b9239aac6ddf400540275a9"} Jul 23 18:23:16.842808 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.842289 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7140746d773d4c171b8d05a5e8fdf9c6aacf3d039b9239aac6ddf400540275a9" Jul 23 18:23:16.844286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.844255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" event={"ID":"0d63f438-3869-4ae6-bed3-e77a7df90a3e","Type":"ContainerDied","Data":"10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793"} Jul 23 18:23:16.844407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.844295 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ac0cdf717abf1e0ecac212ffc10dd4bc9c03f5bf21a0dfa5fc68a0d807f793" Jul 23 18:23:16.844469 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.844413 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2" Jul 23 18:23:16.986925 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:16.986903 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:17.010830 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.010804 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:17.064306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util\") pod \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " Jul 23 18:23:17.064401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064340 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzrnx\" (UniqueName: \"kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx\") pod \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " Jul 23 18:23:17.064401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzfh\" (UniqueName: \"kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh\") pod \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " Jul 23 18:23:17.064401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064398 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle\") pod \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " Jul 23 18:23:17.064536 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064419 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle\") pod \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\" (UID: \"4a7b28d3-ef3d-448c-ab03-16c7c68ba088\") " Jul 23 18:23:17.064536 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064450 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util\") pod \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\" (UID: \"e94fe07c-a82e-4232-90ce-b3fd4731b3b2\") " Jul 23 18:23:17.064943 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.064919 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle" (OuterVolumeSpecName: "bundle") pod "4a7b28d3-ef3d-448c-ab03-16c7c68ba088" (UID: "4a7b28d3-ef3d-448c-ab03-16c7c68ba088"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:17.065155 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.065126 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle" (OuterVolumeSpecName: "bundle") pod "e94fe07c-a82e-4232-90ce-b3fd4731b3b2" (UID: "e94fe07c-a82e-4232-90ce-b3fd4731b3b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:17.066567 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.066541 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx" (OuterVolumeSpecName: "kube-api-access-jzrnx") pod "e94fe07c-a82e-4232-90ce-b3fd4731b3b2" (UID: "e94fe07c-a82e-4232-90ce-b3fd4731b3b2"). InnerVolumeSpecName "kube-api-access-jzrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:23:17.066667 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.066642 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh" (OuterVolumeSpecName: "kube-api-access-4wzfh") pod "4a7b28d3-ef3d-448c-ab03-16c7c68ba088" (UID: "4a7b28d3-ef3d-448c-ab03-16c7c68ba088"). InnerVolumeSpecName "kube-api-access-4wzfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:23:17.070105 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.070084 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util" (OuterVolumeSpecName: "util") pod "4a7b28d3-ef3d-448c-ab03-16c7c68ba088" (UID: "4a7b28d3-ef3d-448c-ab03-16c7c68ba088"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:17.070359 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.070341 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util" (OuterVolumeSpecName: "util") pod "e94fe07c-a82e-4232-90ce-b3fd4731b3b2" (UID: "e94fe07c-a82e-4232-90ce-b3fd4731b3b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:23:17.165918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165866 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.165918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165888 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.165918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165896 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.165918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165903 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-util\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.165918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165911 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrnx\" (UniqueName: \"kubernetes.io/projected/e94fe07c-a82e-4232-90ce-b3fd4731b3b2-kube-api-access-jzrnx\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.166102 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.165921 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzfh\" (UniqueName: \"kubernetes.io/projected/4a7b28d3-ef3d-448c-ab03-16c7c68ba088-kube-api-access-4wzfh\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:23:17.851396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.851360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" Jul 23 18:23:17.851783 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.851361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd" event={"ID":"4a7b28d3-ef3d-448c-ab03-16c7c68ba088","Type":"ContainerDied","Data":"a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a"} Jul 23 18:23:17.851783 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.851504 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b3bd120aaaf58568dc2ded650a459854f6215fad56720f288951e4f0d8a21a" Jul 23 18:23:17.853747 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.853714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" event={"ID":"e94fe07c-a82e-4232-90ce-b3fd4731b3b2","Type":"ContainerDied","Data":"ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c"} Jul 23 18:23:17.853871 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.853761 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff90b5a821e7b7b99be5a00b53b2fe1fb22628027a5073918bbb0d49d159774c" Jul 23 18:23:17.853871 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:17.853794 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv" Jul 23 18:23:19.405244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:19.405197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:22.442927 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:22.442889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:24.829652 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:24.829620 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:24.841130 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:24.841103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:26.396913 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:26.396860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:26.804550 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:26.804520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:27.462374 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:27.462336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:28.381691 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:28.381647 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:28.381897 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:28.381823 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:29.124482 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:29.124436 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:29.209862 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.209822 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg"] Jul 23 18:23:29.210182 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210167 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="util" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="util" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="pull" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="pull" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210207 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="extract" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210213 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="extract" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210237 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="util" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210246 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="util" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210253 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="extract" Jul 23 18:23:29.210256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210259 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210270 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210276 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210284 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210290 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210295 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210302 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210317 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="util" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210324 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="util" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210331 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210337 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="pull" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210349 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="util" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210355 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="util" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210360 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210366 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210421 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94fe07c-a82e-4232-90ce-b3fd4731b3b2" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210428 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a7b28d3-ef3d-448c-ab03-16c7c68ba088" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210436 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d63f438-3869-4ae6-bed3-e77a7df90a3e" containerName="extract" Jul 23 18:23:29.210510 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.210444 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec25ebc5-75f5-4022-a96e-58091705762e" containerName="extract" Jul 23 18:23:29.214668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.214648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:29.217522 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.217497 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 18:23:29.218177 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.218158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6msqq\"" Jul 23 18:23:29.218311 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.218162 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 18:23:29.224434 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.224409 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg"] Jul 23 18:23:29.354341 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.354303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnft\" (UniqueName: \"kubernetes.io/projected/271f97f7-ae58-481d-a499-982f72f63c01-kube-api-access-6lnft\") pod \"authorino-operator-75cc8d5cb7-m8ddg\" (UID: \"271f97f7-ae58-481d-a499-982f72f63c01\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:29.455059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.455023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnft\" (UniqueName: \"kubernetes.io/projected/271f97f7-ae58-481d-a499-982f72f63c01-kube-api-access-6lnft\") pod \"authorino-operator-75cc8d5cb7-m8ddg\" (UID: \"271f97f7-ae58-481d-a499-982f72f63c01\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:29.469901 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.469873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnft\" (UniqueName: \"kubernetes.io/projected/271f97f7-ae58-481d-a499-982f72f63c01-kube-api-access-6lnft\") pod \"authorino-operator-75cc8d5cb7-m8ddg\" (UID: \"271f97f7-ae58-481d-a499-982f72f63c01\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:29.525777 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.525755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:29.658798 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.658774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg"] Jul 23 18:23:29.659775 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:23:29.659750 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271f97f7_ae58_481d_a499_982f72f63c01.slice/crio-054a3bddb6c8cd0308f27ae38ebeeefd5823053b3f86f0201c3112bde66d65cd WatchSource:0}: Error finding container 054a3bddb6c8cd0308f27ae38ebeeefd5823053b3f86f0201c3112bde66d65cd: Status 404 returned error can't find the container with id 054a3bddb6c8cd0308f27ae38ebeeefd5823053b3f86f0201c3112bde66d65cd Jul 23 18:23:29.897139 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.897041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:29.917374 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:29.917346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" event={"ID":"271f97f7-ae58-481d-a499-982f72f63c01","Type":"ContainerStarted","Data":"054a3bddb6c8cd0308f27ae38ebeeefd5823053b3f86f0201c3112bde66d65cd"} Jul 23 18:23:30.008606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:30.008563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:30.131045 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:30.131019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:31.638373 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:31.638342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:32.935244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.935195 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" event={"ID":"271f97f7-ae58-481d-a499-982f72f63c01","Type":"ContainerStarted","Data":"a2f9cea616d23444695c2f283887a873b5eea294dafc1c73fa9c263b2bb7a03f"} Jul 23 18:23:32.935639 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.935375 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:32.955161 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.955131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jq259"] Jul 23 18:23:32.958518 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.958499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:32.959786 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.959743 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" podStartSLOduration=1.665386844 podStartE2EDuration="3.95973178s" totalImagesPullingTime="2.294344936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:32.957459843 +0000 UTC m=+439.688134956" watchObservedRunningTime="2026-07-23 18:23:32.95973178 +0000 UTC m=+439.690406939" Jul 23 18:23:32.960924 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.960890 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 23 18:23:32.961122 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.961106 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 23 18:23:32.961200 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.961186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w6tlp\"" Jul 23 18:23:32.970262 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.970241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jq259"] Jul 23 18:23:32.976725 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:32.976703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:33.087273 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.087245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.087413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.087277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e56ada6-6109-48e0-9914-98c7698577bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.087413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.087309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxgj\" (UniqueName: \"kubernetes.io/projected/6e56ada6-6109-48e0-9914-98c7698577bc-kube-api-access-2hxgj\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.187714 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.187649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.187714 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.187679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e56ada6-6109-48e0-9914-98c7698577bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.187714 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.187696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxgj\" (UniqueName: \"kubernetes.io/projected/6e56ada6-6109-48e0-9914-98c7698577bc-kube-api-access-2hxgj\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.187882 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:33.187782 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 23 18:23:33.187882 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:33.187853 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert podName:6e56ada6-6109-48e0-9914-98c7698577bc nodeName:}" failed. No retries permitted until 2026-07-23 18:23:33.687836201 +0000 UTC m=+440.418511285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jq259" (UID: "6e56ada6-6109-48e0-9914-98c7698577bc") : secret "plugin-serving-cert" not found Jul 23 18:23:33.188317 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.188301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e56ada6-6109-48e0-9914-98c7698577bc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.203210 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.203189 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxgj\" (UniqueName: \"kubernetes.io/projected/6e56ada6-6109-48e0-9914-98c7698577bc-kube-api-access-2hxgj\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.564583 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.564545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:33.691199 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.691170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.693645 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.693625 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e56ada6-6109-48e0-9914-98c7698577bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jq259\" (UID: \"6e56ada6-6109-48e0-9914-98c7698577bc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:33.867603 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:33.867535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" Jul 23 18:23:34.012185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:34.012161 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jq259"] Jul 23 18:23:34.014378 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:23:34.014350 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e56ada6_6109_48e0_9914_98c7698577bc.slice/crio-15b22e894f4accac15266432ad150d9e726e00938dbde4470c037a32069b85d4 WatchSource:0}: Error finding container 15b22e894f4accac15266432ad150d9e726e00938dbde4470c037a32069b85d4: Status 404 returned error can't find the container with id 15b22e894f4accac15266432ad150d9e726e00938dbde4470c037a32069b85d4 Jul 23 18:23:34.414960 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:34.414917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:34.514809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:34.514775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:34.879560 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:34.879516 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:34.947928 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:34.947892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" event={"ID":"6e56ada6-6109-48e0-9914-98c7698577bc","Type":"ContainerStarted","Data":"15b22e894f4accac15266432ad150d9e726e00938dbde4470c037a32069b85d4"} Jul 23 18:23:42.621921 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:42.621882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:42.725146 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:42.725098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:43.944784 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:43.944747 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m8ddg" Jul 23 18:23:44.157498 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:44.157461 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:44.890198 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:44.890156 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:47.491740 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:47.491703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:47.586561 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:47.586496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:49.882624 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:49.882575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:49.952869 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:49.952828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:54.939930 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:54.939892 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:23:55.639852 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:55.639809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:55.739982 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:55.739943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:23:58.083557 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:58.083515 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" event={"ID":"6e56ada6-6109-48e0-9914-98c7698577bc","Type":"ContainerStarted","Data":"90d3d6861e93f7f84fdaea7233698f61fb45b32c924f781a89e935f4369af500"} Jul 23 18:23:58.101644 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:23:58.101588 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jq259" podStartSLOduration=2.78814364 podStartE2EDuration="26.101574152s" totalImagesPullingTime="23.313430512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:23:58.098504062 +0000 UTC m=+464.829179154" watchObservedRunningTime="2026-07-23 18:23:58.101574152 +0000 UTC m=+464.832249243" Jul 23 18:23:59.123176 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:23:59.123139 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:24:03.447847 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:03.447811 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:03.549885 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:03.549848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:04.982835 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:24:04.982804 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d63f438_3869_4ae6_bed3_e77a7df90a3e.slice/crio-conmon-cd41941fff4cd58c1ac75c1ac59a619e8402ea20d6f5b8838f6b924f3db50639.scope\": RecentStats: unable to find data in memory cache]" Jul 23 18:24:07.873791 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:07.873747 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:07.970148 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:07.970109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:08.046565 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:08.046526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:10.416925 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:10.416887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:10.512090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:10.512059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:11.966908 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:11.966869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:12.107810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:12.107771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:13.713947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:13.713916 2576 scope.go:117] "RemoveContainer" containerID="834ba02132170fa1c1df8bfc5226a7abfb096ccb716d5cb563baf7af75f56727" Jul 23 18:24:15.656685 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:15.656637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:15.760455 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:15.760419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:16.686634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:16.686603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:16.823314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:16.823281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:16.894047 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:16.893992 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:17.557546 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.557511 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:17.561212 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.561172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:17.563644 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.563622 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w9nwc\"" Jul 23 18:24:17.569695 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.569675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:17.591881 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.591854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:17.653993 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.653963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:17.668021 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.667995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6ck\" (UniqueName: \"kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck\") pod \"authorino-58cd955675-v4ml2\" (UID: \"09f626df-c51a-4e78-9602-28a07cee893e\") " pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:17.684283 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.684260 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:17.686475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.686460 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:17.697083 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.697052 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:17.769400 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.769370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nx6\" (UniqueName: \"kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6\") pod \"authorino-75d556bd77-hbth8\" (UID: \"820a0eae-d8cd-4707-ac55-b2542f085d6a\") " pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:17.769516 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.769432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6ck\" (UniqueName: \"kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck\") pod \"authorino-58cd955675-v4ml2\" (UID: \"09f626df-c51a-4e78-9602-28a07cee893e\") " pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:17.779048 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.779024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6ck\" (UniqueName: \"kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck\") pod \"authorino-58cd955675-v4ml2\" (UID: \"09f626df-c51a-4e78-9602-28a07cee893e\") " pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:17.870210 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.870130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:17.870399 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.870378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nx6\" (UniqueName: \"kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6\") pod \"authorino-75d556bd77-hbth8\" (UID: \"820a0eae-d8cd-4707-ac55-b2542f085d6a\") " pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:17.878994 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.878969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nx6\" (UniqueName: \"kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6\") pod \"authorino-75d556bd77-hbth8\" (UID: \"820a0eae-d8cd-4707-ac55-b2542f085d6a\") " pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:17.994726 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:17.994693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:18.005088 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:18.005056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:18.007825 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:24:18.007801 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09f626df_c51a_4e78_9602_28a07cee893e.slice/crio-bce96f681f7b6528d391d78bfb6a66cc1e898364d8e8b801c6ced4acd6ce3e4a WatchSource:0}: Error finding container bce96f681f7b6528d391d78bfb6a66cc1e898364d8e8b801c6ced4acd6ce3e4a: Status 404 returned error can't find the container with id bce96f681f7b6528d391d78bfb6a66cc1e898364d8e8b801c6ced4acd6ce3e4a Jul 23 18:24:18.134553 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:18.134529 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:18.136634 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:24:18.136611 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod820a0eae_d8cd_4707_ac55_b2542f085d6a.slice/crio-46004c2307c0487c28a179dc226d69e0ff90d1a7dcf89b8cb1acd52f8ff2929c WatchSource:0}: Error finding container 46004c2307c0487c28a179dc226d69e0ff90d1a7dcf89b8cb1acd52f8ff2929c: Status 404 returned error can't find the container with id 46004c2307c0487c28a179dc226d69e0ff90d1a7dcf89b8cb1acd52f8ff2929c Jul 23 18:24:18.183589 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:18.183558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbth8" event={"ID":"820a0eae-d8cd-4707-ac55-b2542f085d6a","Type":"ContainerStarted","Data":"46004c2307c0487c28a179dc226d69e0ff90d1a7dcf89b8cb1acd52f8ff2929c"} Jul 23 18:24:18.184744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:18.184720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v4ml2" event={"ID":"09f626df-c51a-4e78-9602-28a07cee893e","Type":"ContainerStarted","Data":"bce96f681f7b6528d391d78bfb6a66cc1e898364d8e8b801c6ced4acd6ce3e4a"} Jul 23 18:24:22.022022 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:22.021989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:23.218580 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:23.218538 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v4ml2" event={"ID":"09f626df-c51a-4e78-9602-28a07cee893e","Type":"ContainerStarted","Data":"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28"} Jul 23 18:24:23.220032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:23.220007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbth8" event={"ID":"820a0eae-d8cd-4707-ac55-b2542f085d6a","Type":"ContainerStarted","Data":"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7"} Jul 23 18:24:23.235026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:23.234983 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-v4ml2" podStartSLOduration=2.091112995 podStartE2EDuration="6.234967186s" totalImagesPullingTime="4.143854191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:23.233752928 +0000 UTC m=+489.964428050" watchObservedRunningTime="2026-07-23 18:24:23.234967186 +0000 UTC m=+489.965642277" Jul 23 18:24:23.249291 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:23.249216 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-hbth8" podStartSLOduration=2.222123466 podStartE2EDuration="6.249197082s" totalImagesPullingTime="4.027073616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:23.247847116 +0000 UTC m=+489.978522237" watchObservedRunningTime="2026-07-23 18:24:23.249197082 +0000 UTC m=+489.979872174" Jul 23 18:24:23.272638 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:23.272598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:24.408990 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.408953 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76c6d65d8d-x76tf"] Jul 23 18:24:24.474574 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.474540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:24.578762 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.578727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:24.677040 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.676962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:24.746947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.746907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c6d65d8d-x76tf"] Jul 23 18:24:24.747099 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.747023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.749077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.749044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:24.837867 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.837837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838017 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.837882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-oauth-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838017 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.837938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-trusted-ca-bundle\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838017 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.837957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-oauth-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.838020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-service-ca\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.838062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-console-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.838379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.838078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2pr\" (UniqueName: \"kubernetes.io/projected/4c8930c4-753e-48b6-8674-9b86936bbfea-kube-api-access-bf2pr\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.938957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.938879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-service-ca\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.938957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.938923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-console-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.938957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.938940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2pr\" (UniqueName: \"kubernetes.io/projected/4c8930c4-753e-48b6-8674-9b86936bbfea-kube-api-access-bf2pr\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939250 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939331 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-oauth-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939395 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-trusted-ca-bundle\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939522 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-oauth-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-service-ca\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.939893 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.939778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-console-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.940135 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.940113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-oauth-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.940268 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.940180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c8930c4-753e-48b6-8674-9b86936bbfea-trusted-ca-bundle\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.942092 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.942073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-oauth-config\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.942245 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.942195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c8930c4-753e-48b6-8674-9b86936bbfea-console-serving-cert\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:24.949366 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:24.949322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2pr\" (UniqueName: \"kubernetes.io/projected/4c8930c4-753e-48b6-8674-9b86936bbfea-kube-api-access-bf2pr\") pod \"console-76c6d65d8d-x76tf\" (UID: \"4c8930c4-753e-48b6-8674-9b86936bbfea\") " pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:25.056379 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.056344 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:25.205684 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.205654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:25.207059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.207040 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c6d65d8d-x76tf"] Jul 23 18:24:25.208788 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:24:25.208750 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8930c4_753e_48b6_8674_9b86936bbfea.slice/crio-4c06ce89d8e899809f65e962cd08df33a2fc810f9956e294e271373d1f9cbf0f WatchSource:0}: Error finding container 4c06ce89d8e899809f65e962cd08df33a2fc810f9956e294e271373d1f9cbf0f: Status 404 returned error can't find the container with id 4c06ce89d8e899809f65e962cd08df33a2fc810f9956e294e271373d1f9cbf0f Jul 23 18:24:25.232893 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.232865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c6d65d8d-x76tf" event={"ID":"4c8930c4-753e-48b6-8674-9b86936bbfea","Type":"ContainerStarted","Data":"4c06ce89d8e899809f65e962cd08df33a2fc810f9956e294e271373d1f9cbf0f"} Jul 23 18:24:25.233039 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.233020 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-v4ml2" podUID="09f626df-c51a-4e78-9602-28a07cee893e" containerName="authorino" containerID="cri-o://fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28" gracePeriod=30 Jul 23 18:24:25.566847 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.566811 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:25.645669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.645640 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr6ck\" (UniqueName: \"kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck\") pod \"09f626df-c51a-4e78-9602-28a07cee893e\" (UID: \"09f626df-c51a-4e78-9602-28a07cee893e\") " Jul 23 18:24:25.647616 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.647594 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck" (OuterVolumeSpecName: "kube-api-access-zr6ck") pod "09f626df-c51a-4e78-9602-28a07cee893e" (UID: "09f626df-c51a-4e78-9602-28a07cee893e"). InnerVolumeSpecName "kube-api-access-zr6ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:25.746856 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:25.746784 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6ck\" (UniqueName: \"kubernetes.io/projected/09f626df-c51a-4e78-9602-28a07cee893e-kube-api-access-zr6ck\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:24:26.239338 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.239303 2576 generic.go:356] "Generic (PLEG): container finished" podID="09f626df-c51a-4e78-9602-28a07cee893e" containerID="fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28" exitCode=0 Jul 23 18:24:26.239519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.239341 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v4ml2" Jul 23 18:24:26.239519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.239393 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v4ml2" event={"ID":"09f626df-c51a-4e78-9602-28a07cee893e","Type":"ContainerDied","Data":"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28"} Jul 23 18:24:26.239519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.239432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v4ml2" event={"ID":"09f626df-c51a-4e78-9602-28a07cee893e","Type":"ContainerDied","Data":"bce96f681f7b6528d391d78bfb6a66cc1e898364d8e8b801c6ced4acd6ce3e4a"} Jul 23 18:24:26.239519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.239449 2576 scope.go:117] "RemoveContainer" containerID="fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28" Jul 23 18:24:26.247614 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.247586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c6d65d8d-x76tf" event={"ID":"4c8930c4-753e-48b6-8674-9b86936bbfea","Type":"ContainerStarted","Data":"8a8e1b4ba730015f3e13691186dd12f387b378d3d889a74d278058b6de9a99b1"} Jul 23 18:24:26.254238 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.254212 2576 scope.go:117] "RemoveContainer" containerID="fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28" Jul 23 18:24:26.254558 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:24:26.254536 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28\": container with ID starting with fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28 not found: ID does not exist" containerID="fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28" Jul 23 18:24:26.254630 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.254564 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28"} err="failed to get container status \"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28\": rpc error: code = NotFound desc = could not find container \"fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28\": container with ID starting with fb2dd9206e16f04b9af2a4af1da6513fe7406fd66663892469acb62ceb0a8a28 not found: ID does not exist" Jul 23 18:24:26.261917 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.261894 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:26.265571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.265542 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-v4ml2"] Jul 23 18:24:26.282525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:26.282488 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76c6d65d8d-x76tf" podStartSLOduration=2.282475764 podStartE2EDuration="2.282475764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:26.28146894 +0000 UTC m=+493.012144033" watchObservedRunningTime="2026-07-23 18:24:26.282475764 +0000 UTC m=+493.013150855" Jul 23 18:24:27.834699 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:27.834668 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f626df-c51a-4e78-9602-28a07cee893e" path="/var/lib/kubelet/pods/09f626df-c51a-4e78-9602-28a07cee893e/volumes" Jul 23 18:24:35.056488 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.056455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:35.056951 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.056506 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:35.062623 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.062600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:35.301383 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.301356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76c6d65d8d-x76tf" Jul 23 18:24:35.352949 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.352873 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:24:35.470714 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.470679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:35.704894 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.704855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:35.806388 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:35.806348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:38.009536 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:38.009501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:38.108693 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:38.108662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:38.473280 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:38.473242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:51.063185 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.063146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:51.300849 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.300816 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sqxp8"] Jul 23 18:24:51.301165 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.301152 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f626df-c51a-4e78-9602-28a07cee893e" containerName="authorino" Jul 23 18:24:51.301214 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.301167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f626df-c51a-4e78-9602-28a07cee893e" containerName="authorino" Jul 23 18:24:51.301294 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.301218 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f626df-c51a-4e78-9602-28a07cee893e" containerName="authorino" Jul 23 18:24:51.303149 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.303133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:51.310031 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.309723 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sqxp8"] Jul 23 18:24:51.407281 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.407192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:51.454170 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.454139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlt8w\" (UniqueName: \"kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w\") pod \"authorino-5d4b67c446-sqxp8\" (UID: \"d3e25ae5-2629-4dec-919f-b9899953c54c\") " pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:51.535972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.535948 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sqxp8"] Jul 23 18:24:51.536159 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:24:51.536141 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rlt8w], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-sqxp8" podUID="d3e25ae5-2629-4dec-919f-b9899953c54c" Jul 23 18:24:51.555121 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.555094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlt8w\" (UniqueName: \"kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w\") pod \"authorino-5d4b67c446-sqxp8\" (UID: \"d3e25ae5-2629-4dec-919f-b9899953c54c\") " pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:51.562666 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.562646 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d4d577f64-pxm8g"] Jul 23 18:24:51.564860 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.564843 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:51.566525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.566505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlt8w\" (UniqueName: \"kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w\") pod \"authorino-5d4b67c446-sqxp8\" (UID: \"d3e25ae5-2629-4dec-919f-b9899953c54c\") " pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:51.576994 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.576966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d4d577f64-pxm8g"] Jul 23 18:24:51.650995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.650965 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4d577f64-pxm8g"] Jul 23 18:24:51.651314 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:24:51.651289 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9jgvx], unattached volumes=[], failed to process volumes=[kube-api-access-9jgvx]: context canceled" pod="kuadrant-system/authorino-d4d577f64-pxm8g" podUID="783d3bfc-d01b-4ae2-91ed-995d40c74b90" Jul 23 18:24:51.681822 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.681757 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:24:51.683910 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.683895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.686074 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.686055 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 18:24:51.691913 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.691895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:24:51.756824 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.756797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgvx\" (UniqueName: \"kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx\") pod \"authorino-d4d577f64-pxm8g\" (UID: \"783d3bfc-d01b-4ae2-91ed-995d40c74b90\") " pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:51.857205 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.857175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x5m\" (UniqueName: \"kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.857391 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.857245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgvx\" (UniqueName: \"kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx\") pod \"authorino-d4d577f64-pxm8g\" (UID: \"783d3bfc-d01b-4ae2-91ed-995d40c74b90\") " pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:51.857391 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.857283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.866413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.866394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgvx\" (UniqueName: \"kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx\") pod \"authorino-d4d577f64-pxm8g\" (UID: \"783d3bfc-d01b-4ae2-91ed-995d40c74b90\") " pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:51.958593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.958568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.958719 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.958630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x5m\" (UniqueName: \"kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.960982 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.960956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.966177 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.966157 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x5m\" (UniqueName: \"kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m\") pod \"authorino-5f64b76ccb-ws6sq\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:51.993633 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:51.993613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:24:52.120181 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.120158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:24:52.121953 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:24:52.121927 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11d049a_2bcc_4611_a738_068426655829.slice/crio-f23580a731ae1d5968d38c865657732a4f772b30570766a26f1092657e5f2be4 WatchSource:0}: Error finding container f23580a731ae1d5968d38c865657732a4f772b30570766a26f1092657e5f2be4: Status 404 returned error can't find the container with id f23580a731ae1d5968d38c865657732a4f772b30570766a26f1092657e5f2be4 Jul 23 18:24:52.381051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.380972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" event={"ID":"a11d049a-2bcc-4611-a738-068426655829","Type":"ContainerStarted","Data":"f23580a731ae1d5968d38c865657732a4f772b30570766a26f1092657e5f2be4"} Jul 23 18:24:52.381051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.381002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:52.381051 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.380989 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:52.387533 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.387509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:52.389887 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.389870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:52.563514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.563476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jgvx\" (UniqueName: \"kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx\") pod \"783d3bfc-d01b-4ae2-91ed-995d40c74b90\" (UID: \"783d3bfc-d01b-4ae2-91ed-995d40c74b90\") " Jul 23 18:24:52.563704 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.563551 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlt8w\" (UniqueName: \"kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w\") pod \"d3e25ae5-2629-4dec-919f-b9899953c54c\" (UID: \"d3e25ae5-2629-4dec-919f-b9899953c54c\") " Jul 23 18:24:52.565702 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.565672 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w" (OuterVolumeSpecName: "kube-api-access-rlt8w") pod "d3e25ae5-2629-4dec-919f-b9899953c54c" (UID: "d3e25ae5-2629-4dec-919f-b9899953c54c"). InnerVolumeSpecName "kube-api-access-rlt8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:52.566060 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.566038 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx" (OuterVolumeSpecName: "kube-api-access-9jgvx") pod "783d3bfc-d01b-4ae2-91ed-995d40c74b90" (UID: "783d3bfc-d01b-4ae2-91ed-995d40c74b90"). InnerVolumeSpecName "kube-api-access-9jgvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:52.664293 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.664184 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgvx\" (UniqueName: \"kubernetes.io/projected/783d3bfc-d01b-4ae2-91ed-995d40c74b90-kube-api-access-9jgvx\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:24:52.664293 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:52.664217 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlt8w\" (UniqueName: \"kubernetes.io/projected/d3e25ae5-2629-4dec-919f-b9899953c54c-kube-api-access-rlt8w\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:24:53.387027 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.386991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" event={"ID":"a11d049a-2bcc-4611-a738-068426655829","Type":"ContainerStarted","Data":"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b"} Jul 23 18:24:53.387467 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.387065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d4d577f64-pxm8g" Jul 23 18:24:53.387467 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.387251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sqxp8" Jul 23 18:24:53.402112 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.402072 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" podStartSLOduration=2.078503112 podStartE2EDuration="2.402057777s" totalImagesPullingTime="323.554665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:24:53.400849005 +0000 UTC m=+520.131524099" watchObservedRunningTime="2026-07-23 18:24:53.402057777 +0000 UTC m=+520.132732867" Jul 23 18:24:53.428979 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.428950 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d4d577f64-pxm8g"] Jul 23 18:24:53.437593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.436872 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:53.437593 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.437069 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-hbth8" podUID="820a0eae-d8cd-4707-ac55-b2542f085d6a" containerName="authorino" containerID="cri-o://2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7" gracePeriod=30 Jul 23 18:24:53.439421 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.439401 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d4d577f64-pxm8g"] Jul 23 18:24:53.461774 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.461740 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sqxp8"] Jul 23 18:24:53.464265 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.464242 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sqxp8"] Jul 23 18:24:53.685574 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.685550 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:53.772407 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.772378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7nx6\" (UniqueName: \"kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6\") pod \"820a0eae-d8cd-4707-ac55-b2542f085d6a\" (UID: \"820a0eae-d8cd-4707-ac55-b2542f085d6a\") " Jul 23 18:24:53.774440 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.774416 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6" (OuterVolumeSpecName: "kube-api-access-s7nx6") pod "820a0eae-d8cd-4707-ac55-b2542f085d6a" (UID: "820a0eae-d8cd-4707-ac55-b2542f085d6a"). InnerVolumeSpecName "kube-api-access-s7nx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:24:53.835265 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.835212 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783d3bfc-d01b-4ae2-91ed-995d40c74b90" path="/var/lib/kubelet/pods/783d3bfc-d01b-4ae2-91ed-995d40c74b90/volumes" Jul 23 18:24:53.835525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.835507 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e25ae5-2629-4dec-919f-b9899953c54c" path="/var/lib/kubelet/pods/d3e25ae5-2629-4dec-919f-b9899953c54c/volumes" Jul 23 18:24:53.872855 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:53.872834 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nx6\" (UniqueName: \"kubernetes.io/projected/820a0eae-d8cd-4707-ac55-b2542f085d6a-kube-api-access-s7nx6\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:24:54.393318 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.393280 2576 generic.go:356] "Generic (PLEG): container finished" podID="820a0eae-d8cd-4707-ac55-b2542f085d6a" containerID="2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7" exitCode=0 Jul 23 18:24:54.393318 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.393318 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hbth8" Jul 23 18:24:54.393837 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.393345 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbth8" event={"ID":"820a0eae-d8cd-4707-ac55-b2542f085d6a","Type":"ContainerDied","Data":"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7"} Jul 23 18:24:54.393837 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.393385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hbth8" event={"ID":"820a0eae-d8cd-4707-ac55-b2542f085d6a","Type":"ContainerDied","Data":"46004c2307c0487c28a179dc226d69e0ff90d1a7dcf89b8cb1acd52f8ff2929c"} Jul 23 18:24:54.393837 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.393404 2576 scope.go:117] "RemoveContainer" containerID="2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7" Jul 23 18:24:54.401946 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.401927 2576 scope.go:117] "RemoveContainer" containerID="2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7" Jul 23 18:24:54.402186 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:24:54.402169 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7\": container with ID starting with 2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7 not found: ID does not exist" containerID="2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7" Jul 23 18:24:54.402256 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.402198 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7"} err="failed to get container status \"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7\": rpc error: code = NotFound desc = could not find container \"2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7\": container with ID starting with 2d9bf69a14e9cd6ecaeae0f51a59d015d8bb291b36f96521511ba43c68f141c7 not found: ID does not exist" Jul 23 18:24:54.408480 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.408457 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:54.411738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:54.411717 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hbth8"] Jul 23 18:24:55.834561 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:55.834526 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820a0eae-d8cd-4707-ac55-b2542f085d6a" path="/var/lib/kubelet/pods/820a0eae-d8cd-4707-ac55-b2542f085d6a/volumes" Jul 23 18:24:58.158634 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:58.158580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:58.256027 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:58.255984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:58.319552 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:58.319523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:24:58.728007 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:24:58.727970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:00.374870 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.374822 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69d9f9df64-p5wzn" podUID="6883de29-c53a-4f88-a2bc-bef51555282b" containerName="console" containerID="cri-o://48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1" gracePeriod=15 Jul 23 18:25:00.621191 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.621168 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:25:00.719803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.719781 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wctf\" (UniqueName: \"kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.719954 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.719841 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.719954 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.719860 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.719954 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.719880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.720125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.719995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.720125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720037 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.720125 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert\") pod \"6883de29-c53a-4f88-a2bc-bef51555282b\" (UID: \"6883de29-c53a-4f88-a2bc-bef51555282b\") " Jul 23 18:25:00.720373 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720345 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config" (OuterVolumeSpecName: "console-config") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:25:00.720460 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720373 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:25:00.720517 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720423 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca" (OuterVolumeSpecName: "service-ca") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:25:00.720571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.720555 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:25:00.721968 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.721937 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf" (OuterVolumeSpecName: "kube-api-access-5wctf") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "kube-api-access-5wctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:25:00.722089 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.722071 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:25:00.722150 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.722098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6883de29-c53a-4f88-a2bc-bef51555282b" (UID: "6883de29-c53a-4f88-a2bc-bef51555282b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:25:00.821346 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821321 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-console-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821346 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821342 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-oauth-config\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821346 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821351 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-trusted-ca-bundle\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821361 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-service-ca\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821369 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6883de29-c53a-4f88-a2bc-bef51555282b-console-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821378 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6883de29-c53a-4f88-a2bc-bef51555282b-oauth-serving-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:00.821525 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:00.821387 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wctf\" (UniqueName: \"kubernetes.io/projected/6883de29-c53a-4f88-a2bc-bef51555282b-kube-api-access-5wctf\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:25:01.430092 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.430061 2576 generic.go:356] "Generic (PLEG): container finished" podID="6883de29-c53a-4f88-a2bc-bef51555282b" containerID="48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1" exitCode=0 Jul 23 18:25:01.430535 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.430106 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d9f9df64-p5wzn" Jul 23 18:25:01.430535 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.430126 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d9f9df64-p5wzn" event={"ID":"6883de29-c53a-4f88-a2bc-bef51555282b","Type":"ContainerDied","Data":"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1"} Jul 23 18:25:01.430535 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.430165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d9f9df64-p5wzn" event={"ID":"6883de29-c53a-4f88-a2bc-bef51555282b","Type":"ContainerDied","Data":"dad2497a47e47c7eed63f8d370c66320fc7adb7e41c2c752006ae2a8120f78f0"} Jul 23 18:25:01.430535 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.430188 2576 scope.go:117] "RemoveContainer" containerID="48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1" Jul 23 18:25:01.439262 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.439220 2576 scope.go:117] "RemoveContainer" containerID="48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1" Jul 23 18:25:01.439536 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:25:01.439518 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1\": container with ID starting with 48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1 not found: ID does not exist" containerID="48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1" Jul 23 18:25:01.439582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.439561 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1"} err="failed to get container status \"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1\": rpc error: code = NotFound desc = could not find container \"48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1\": container with ID starting with 48f72b4b2351dab9b85456158d4a1b4d3df35a98df8740bbb7850feaaf50a1c1 not found: ID does not exist" Jul 23 18:25:01.451999 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.451972 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:25:01.455551 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.455531 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d9f9df64-p5wzn"] Jul 23 18:25:01.834316 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.834275 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6883de29-c53a-4f88-a2bc-bef51555282b" path="/var/lib/kubelet/pods/6883de29-c53a-4f88-a2bc-bef51555282b/volumes" Jul 23 18:25:01.984454 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:01.984411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:02.084278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:02.084248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:05.485371 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:05.485279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:05.536524 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:05.536493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:18.280013 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:18.279977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:18.334375 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:18.334345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:33.637411 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:33.637377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:33.735138 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:33.735097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:34.306406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.306373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:34.772694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.772660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:34.862963 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.862929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:34.899844 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.899812 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vm7l2"] Jul 23 18:25:34.900162 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900149 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6883de29-c53a-4f88-a2bc-bef51555282b" containerName="console" Jul 23 18:25:34.900207 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900165 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6883de29-c53a-4f88-a2bc-bef51555282b" containerName="console" Jul 23 18:25:34.900207 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900177 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820a0eae-d8cd-4707-ac55-b2542f085d6a" containerName="authorino" Jul 23 18:25:34.900207 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="820a0eae-d8cd-4707-ac55-b2542f085d6a" containerName="authorino" Jul 23 18:25:34.900337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900255 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="820a0eae-d8cd-4707-ac55-b2542f085d6a" containerName="authorino" Jul 23 18:25:34.900337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.900264 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6883de29-c53a-4f88-a2bc-bef51555282b" containerName="console" Jul 23 18:25:34.904529 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.904510 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:34.907236 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.907212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7nbbv\"" Jul 23 18:25:34.907367 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.907220 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 18:25:34.913363 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.913340 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vm7l2"] Jul 23 18:25:34.980184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.980152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1134ff-6dde-4122-bdae-a79bba9cc36e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:34.980362 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:34.980190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6rw\" (UniqueName: \"kubernetes.io/projected/3b1134ff-6dde-4122-bdae-a79bba9cc36e-kube-api-access-jv6rw\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.081033 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.080938 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1134ff-6dde-4122-bdae-a79bba9cc36e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.081033 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.080978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6rw\" (UniqueName: \"kubernetes.io/projected/3b1134ff-6dde-4122-bdae-a79bba9cc36e-kube-api-access-jv6rw\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.081796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.081770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1134ff-6dde-4122-bdae-a79bba9cc36e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.089242 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.089205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6rw\" (UniqueName: \"kubernetes.io/projected/3b1134ff-6dde-4122-bdae-a79bba9cc36e-kube-api-access-jv6rw\") pod \"payload-processing-5bf47cd988-vm7l2\" (UID: \"3b1134ff-6dde-4122-bdae-a79bba9cc36e\") " pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.215766 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.215735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:35.350988 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.350959 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vm7l2"] Jul 23 18:25:35.352868 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:25:35.352836 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1134ff_6dde_4122_bdae_a79bba9cc36e.slice/crio-04bec7f07e744b57577e21ea4ef36d779eded9b8c4e2c33f299fd8ad50eebaf0 WatchSource:0}: Error finding container 04bec7f07e744b57577e21ea4ef36d779eded9b8c4e2c33f299fd8ad50eebaf0: Status 404 returned error can't find the container with id 04bec7f07e744b57577e21ea4ef36d779eded9b8c4e2c33f299fd8ad50eebaf0 Jul 23 18:25:35.606575 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:35.606491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" event={"ID":"3b1134ff-6dde-4122-bdae-a79bba9cc36e","Type":"ContainerStarted","Data":"04bec7f07e744b57577e21ea4ef36d779eded9b8c4e2c33f299fd8ad50eebaf0"} Jul 23 18:25:37.618280 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:37.618249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" event={"ID":"3b1134ff-6dde-4122-bdae-a79bba9cc36e","Type":"ContainerStarted","Data":"1839c17226ae051f7ca86ed2846119c9b60709eeb0b4f90637f40a4dc0cfe3c0"} Jul 23 18:25:37.618653 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:37.618315 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:37.636082 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:37.636033 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" podStartSLOduration=2.215550432 podStartE2EDuration="3.636019771s" totalImagesPullingTime="1.420469339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:25:37.633820498 +0000 UTC m=+564.364495590" watchObservedRunningTime="2026-07-23 18:25:37.636019771 +0000 UTC m=+564.366694862" Jul 23 18:25:44.990690 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:44.990640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:45.089070 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:45.089038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:25:48.626948 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:48.626910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-vm7l2" Jul 23 18:25:59.331144 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:25:59.331090 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:05.138498 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:05.138464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:13.720787 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:13.720747 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:26:13.721415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:13.721386 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:26:13.724950 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:13.724931 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:26:13.725598 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:13.725579 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:26:39.786949 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.786869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd"] Jul 23 18:26:39.790082 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.790066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.792847 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.792825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 18:26:39.793657 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.793636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 23 18:26:39.793755 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.793656 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 18:26:39.793943 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.793929 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7pzzs\"" Jul 23 18:26:39.804105 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.804084 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd"] Jul 23 18:26:39.810895 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.810872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:39.899491 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.899668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.899668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.899668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjhx\" (UniqueName: \"kubernetes.io/projected/bfddb486-1cc9-4985-8c1d-d0c548de9664-kube-api-access-svjhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.899819 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:39.899819 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:39.899786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfddb486-1cc9-4985-8c1d-d0c548de9664-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001068 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfddb486-1cc9-4985-8c1d-d0c548de9664-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001290 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjhx\" (UniqueName: \"kubernetes.io/projected/bfddb486-1cc9-4985-8c1d-d0c548de9664-kube-api-access-svjhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001588 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001651 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.001697 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.001663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.003358 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.003337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfddb486-1cc9-4985-8c1d-d0c548de9664-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.003618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.003600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfddb486-1cc9-4985-8c1d-d0c548de9664-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.010026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.010003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjhx\" (UniqueName: \"kubernetes.io/projected/bfddb486-1cc9-4985-8c1d-d0c548de9664-kube-api-access-svjhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j6rcd\" (UID: \"bfddb486-1cc9-4985-8c1d-d0c548de9664\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.099697 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.099625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:40.236558 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.236532 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd"] Jul 23 18:26:40.238021 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:26:40.237997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfddb486_1cc9_4985_8c1d_d0c548de9664.slice/crio-3b7ed72c5336264314254d0bf0347ad5a6726b032dd83133657a479a1193896b WatchSource:0}: Error finding container 3b7ed72c5336264314254d0bf0347ad5a6726b032dd83133657a479a1193896b: Status 404 returned error can't find the container with id 3b7ed72c5336264314254d0bf0347ad5a6726b032dd83133657a479a1193896b Jul 23 18:26:40.239387 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.239368 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:26:40.942980 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:40.942905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" event={"ID":"bfddb486-1cc9-4985-8c1d-d0c548de9664","Type":"ContainerStarted","Data":"3b7ed72c5336264314254d0bf0347ad5a6726b032dd83133657a479a1193896b"} Jul 23 18:26:41.950258 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:41.950208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" event={"ID":"bfddb486-1cc9-4985-8c1d-d0c548de9664","Type":"ContainerStarted","Data":"5902b9596e517a205f89a892c68225423b4419bb4267b09182d5fa7c7383ad78"} Jul 23 18:26:41.970874 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:41.970819 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" podStartSLOduration=1.812484638 podStartE2EDuration="2.970802354s" totalImagesPullingTime="1.158317716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:26:41.96841342 +0000 UTC m=+628.699088522" watchObservedRunningTime="2026-07-23 18:26:41.970802354 +0000 UTC m=+628.701477445" Jul 23 18:26:46.256532 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:46.256489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:47.697627 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.697592 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v"] Jul 23 18:26:47.701080 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.701057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.703551 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.703528 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 23 18:26:47.714061 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.714035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v"] Jul 23 18:26:47.736743 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.736702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:47.768806 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kcc\" (UniqueName: \"kubernetes.io/projected/9b61e365-5215-44cd-b70b-cc7e140e3024-kube-api-access-94kcc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.768806 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.769010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.769010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.769010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.769010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.768990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b61e365-5215-44cd-b70b-cc7e140e3024-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kcc\" (UniqueName: \"kubernetes.io/projected/9b61e365-5215-44cd-b70b-cc7e140e3024-kube-api-access-94kcc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869580 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869738 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b61e365-5215-44cd-b70b-cc7e140e3024-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.869957 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.869933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.870025 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.870007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.870091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.870076 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.871890 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.871869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b61e365-5215-44cd-b70b-cc7e140e3024-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.872158 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.872140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b61e365-5215-44cd-b70b-cc7e140e3024-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:47.877381 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:47.877363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kcc\" (UniqueName: \"kubernetes.io/projected/9b61e365-5215-44cd-b70b-cc7e140e3024-kube-api-access-94kcc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrm5v\" (UID: \"9b61e365-5215-44cd-b70b-cc7e140e3024\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:48.010905 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:48.010825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:48.141763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:48.141729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v"] Jul 23 18:26:48.144616 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:26:48.144569 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b61e365_5215_44cd_b70b_cc7e140e3024.slice/crio-0828b6924a035a2310250b336620270bb8b34b121308db5d059f6f7b4fb7b84e WatchSource:0}: Error finding container 0828b6924a035a2310250b336620270bb8b34b121308db5d059f6f7b4fb7b84e: Status 404 returned error can't find the container with id 0828b6924a035a2310250b336620270bb8b34b121308db5d059f6f7b4fb7b84e Jul 23 18:26:48.995323 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:48.995274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" event={"ID":"9b61e365-5215-44cd-b70b-cc7e140e3024","Type":"ContainerStarted","Data":"682a8fcb508869ddeab3ef1fe434f20f9d1a879b92529ff5341ee12b1aeba1ca"} Jul 23 18:26:48.995809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:48.995349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" event={"ID":"9b61e365-5215-44cd-b70b-cc7e140e3024","Type":"ContainerStarted","Data":"0828b6924a035a2310250b336620270bb8b34b121308db5d059f6f7b4fb7b84e"} Jul 23 18:26:49.016594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:49.016531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" podStartSLOduration=1.816509104 podStartE2EDuration="2.016509869s" totalImagesPullingTime="200.000765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:26:49.014349631 +0000 UTC m=+635.745024747" watchObservedRunningTime="2026-07-23 18:26:49.016509869 +0000 UTC m=+635.747184979" Jul 23 18:26:50.100683 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:50.100648 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:50.101080 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:50.100712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:50.113365 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:50.113332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:51.016897 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:51.016858 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j6rcd" Jul 23 18:26:57.635992 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:57.635945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:26:58.011261 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:58.011205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:58.011453 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:58.011286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:58.023540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:58.023511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:26:58.053384 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:26:58.053356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrm5v" Jul 23 18:27:21.505158 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.505126 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb"] Jul 23 18:27:21.510505 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.510485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.513039 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.513019 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 18:27:21.521534 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.521514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb"] Jul 23 18:27:21.541801 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.541780 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:27:21.558645 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvkf\" (UniqueName: \"kubernetes.io/projected/4648a87e-6042-4d90-9b75-7a5dabe3aa53-kube-api-access-bsvkf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.558781 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.558781 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.558781 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.558892 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.558892 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.558818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659156 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659301 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659301 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659301 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659301 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659534 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvkf\" (UniqueName: \"kubernetes.io/projected/4648a87e-6042-4d90-9b75-7a5dabe3aa53-kube-api-access-bsvkf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659534 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659650 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.659687 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.659661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.661451 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.661431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4648a87e-6042-4d90-9b75-7a5dabe3aa53-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.661567 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.661549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4648a87e-6042-4d90-9b75-7a5dabe3aa53-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.667691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.667668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvkf\" (UniqueName: \"kubernetes.io/projected/4648a87e-6042-4d90-9b75-7a5dabe3aa53-kube-api-access-bsvkf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nbdvb\" (UID: \"4648a87e-6042-4d90-9b75-7a5dabe3aa53\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.821446 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.821369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:21.956494 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:21.956471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb"] Jul 23 18:27:21.958440 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:27:21.958411 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4648a87e_6042_4d90_9b75_7a5dabe3aa53.slice/crio-2d477f02d75b07ee1de69ea094131f102a6d654876b36cd911159646a731e64f WatchSource:0}: Error finding container 2d477f02d75b07ee1de69ea094131f102a6d654876b36cd911159646a731e64f: Status 404 returned error can't find the container with id 2d477f02d75b07ee1de69ea094131f102a6d654876b36cd911159646a731e64f Jul 23 18:27:22.164637 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:22.164610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" event={"ID":"4648a87e-6042-4d90-9b75-7a5dabe3aa53","Type":"ContainerStarted","Data":"2d477f02d75b07ee1de69ea094131f102a6d654876b36cd911159646a731e64f"} Jul 23 18:27:23.176828 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:23.176789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" event={"ID":"4648a87e-6042-4d90-9b75-7a5dabe3aa53","Type":"ContainerStarted","Data":"d30c251913f8dfb941ed332e28457df677f1142211fe9b6e29c1c9439cc6c28a"} Jul 23 18:27:23.208444 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:23.208396 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" podStartSLOduration=2.037773786 podStartE2EDuration="2.208381917s" totalImagesPullingTime="170.608131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:27:23.20821686 +0000 UTC m=+669.938891953" watchObservedRunningTime="2026-07-23 18:27:23.208381917 +0000 UTC m=+669.939057022" Jul 23 18:27:25.844152 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.844120 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s"] Jul 23 18:27:25.847549 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.847532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.850176 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.850151 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 23 18:27:25.866569 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.866545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s"] Jul 23 18:27:25.896401 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.896556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.896556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76xg\" (UniqueName: \"kubernetes.io/projected/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-kube-api-access-z76xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.896676 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.896676 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.896788 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.896704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.947069 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.947015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:27:25.997911 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.997878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998068 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.997937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998068 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.997972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998068 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998285 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76xg\" (UniqueName: \"kubernetes.io/projected/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-kube-api-access-z76xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998395 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998378 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998473 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998448 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:25.998527 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:25.998505 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:26.000622 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:26.000601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:26.001163 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:26.001138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:26.012986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:26.012963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76xg\" (UniqueName: \"kubernetes.io/projected/b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2-kube-api-access-z76xg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5555s\" (UID: \"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:26.157028 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:26.156933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:26.311755 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:26.311571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s"] Jul 23 18:27:26.314303 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:27:26.314273 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e30616_84a2_4b8e_9dcc_7e625d9b6ba2.slice/crio-1b21559e83e1491284a0dc06c2581bac88aa3a1ae07e04ba92a75d1da9067748 WatchSource:0}: Error finding container 1b21559e83e1491284a0dc06c2581bac88aa3a1ae07e04ba92a75d1da9067748: Status 404 returned error can't find the container with id 1b21559e83e1491284a0dc06c2581bac88aa3a1ae07e04ba92a75d1da9067748 Jul 23 18:27:27.200597 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:27.200556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" event={"ID":"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2","Type":"ContainerStarted","Data":"62122fee45673904877a15d285521ae41e2dc9ffec561dd7f29153c39efc2bb8"} Jul 23 18:27:27.200967 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:27.200606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" event={"ID":"b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2","Type":"ContainerStarted","Data":"1b21559e83e1491284a0dc06c2581bac88aa3a1ae07e04ba92a75d1da9067748"} Jul 23 18:27:27.223330 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:27.223283 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" podStartSLOduration=2.024492247 podStartE2EDuration="2.223266965s" totalImagesPullingTime="198.774718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:27:27.22202586 +0000 UTC m=+673.952700975" watchObservedRunningTime="2026-07-23 18:27:27.223266965 +0000 UTC m=+673.953942056" Jul 23 18:27:31.822541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:31.822510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:31.822912 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:31.822568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:31.837333 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:31.837308 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:32.244143 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:32.244111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nbdvb" Jul 23 18:27:36.157540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:36.157506 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:36.157990 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:36.157551 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:36.169718 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:36.169696 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:36.260841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:36.260816 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5555s" Jul 23 18:27:36.273126 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:27:36.273105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:03.244615 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:03.244583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:11.506730 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:11.506636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:11.636340 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:11.636296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:17.431997 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.431963 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54d49797fc-h5jwv"] Jul 23 18:28:17.435995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.435977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.489363 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.489336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d49797fc-h5jwv"] Jul 23 18:28:17.595981 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.595950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:17.628077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.628053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-tls-cert\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.628179 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.628110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxld\" (UniqueName: \"kubernetes.io/projected/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-kube-api-access-nnxld\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.728639 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.728613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxld\" (UniqueName: \"kubernetes.io/projected/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-kube-api-access-nnxld\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.728740 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.728668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-tls-cert\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.731020 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.731001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-tls-cert\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.747618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.747593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxld\" (UniqueName: \"kubernetes.io/projected/41a8b0d2-6740-40d4-bd35-e5eacaf33d22-kube-api-access-nnxld\") pod \"authorino-54d49797fc-h5jwv\" (UID: \"41a8b0d2-6740-40d4-bd35-e5eacaf33d22\") " pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:17.753105 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:17.753078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:18.046021 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:18.045951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d49797fc-h5jwv" Jul 23 18:28:18.183104 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:18.183078 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d49797fc-h5jwv"] Jul 23 18:28:18.184436 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:28:18.184410 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41a8b0d2_6740_40d4_bd35_e5eacaf33d22.slice/crio-d614e332700e125aab13780395406aa72e1cbb2629f06302e2011ad32266dd2a WatchSource:0}: Error finding container d614e332700e125aab13780395406aa72e1cbb2629f06302e2011ad32266dd2a: Status 404 returned error can't find the container with id d614e332700e125aab13780395406aa72e1cbb2629f06302e2011ad32266dd2a Jul 23 18:28:18.482368 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:18.482340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d49797fc-h5jwv" event={"ID":"41a8b0d2-6740-40d4-bd35-e5eacaf33d22","Type":"ContainerStarted","Data":"d614e332700e125aab13780395406aa72e1cbb2629f06302e2011ad32266dd2a"} Jul 23 18:28:18.842313 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:18.842240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:19.021306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:19.021276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:28:20.496693 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.496644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d49797fc-h5jwv" event={"ID":"41a8b0d2-6740-40d4-bd35-e5eacaf33d22","Type":"ContainerStarted","Data":"1ad774ddf2ffec32122d88470af184bc1e90f83874a910d9bfa241c7953637b3"} Jul 23 18:28:20.541763 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.541709 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54d49797fc-h5jwv" podStartSLOduration=2.068485476 podStartE2EDuration="3.541692428s" totalImagesPullingTime="1.473206952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:28:20.541285899 +0000 UTC m=+727.271961016" watchObservedRunningTime="2026-07-23 18:28:20.541692428 +0000 UTC m=+727.272367518" Jul 23 18:28:20.609731 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.609695 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:28:20.609945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.609916 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" podUID="a11d049a-2bcc-4611-a738-068426655829" containerName="authorino" containerID="cri-o://a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b" gracePeriod=30 Jul 23 18:28:20.883211 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.883187 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:28:20.954794 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.954769 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5x5m\" (UniqueName: \"kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m\") pod \"a11d049a-2bcc-4611-a738-068426655829\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " Jul 23 18:28:20.954954 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.954839 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert\") pod \"a11d049a-2bcc-4611-a738-068426655829\" (UID: \"a11d049a-2bcc-4611-a738-068426655829\") " Jul 23 18:28:20.956872 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.956832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m" (OuterVolumeSpecName: "kube-api-access-g5x5m") pod "a11d049a-2bcc-4611-a738-068426655829" (UID: "a11d049a-2bcc-4611-a738-068426655829"). InnerVolumeSpecName "kube-api-access-g5x5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:28:20.964772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:20.964747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a11d049a-2bcc-4611-a738-068426655829" (UID: "a11d049a-2bcc-4611-a738-068426655829"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:28:21.055632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.055561 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5x5m\" (UniqueName: \"kubernetes.io/projected/a11d049a-2bcc-4611-a738-068426655829-kube-api-access-g5x5m\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:28:21.055632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.055588 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a11d049a-2bcc-4611-a738-068426655829-tls-cert\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:28:21.502871 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.502837 2576 generic.go:356] "Generic (PLEG): container finished" podID="a11d049a-2bcc-4611-a738-068426655829" containerID="a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b" exitCode=0 Jul 23 18:28:21.503351 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.502877 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" Jul 23 18:28:21.503351 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.502930 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" event={"ID":"a11d049a-2bcc-4611-a738-068426655829","Type":"ContainerDied","Data":"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b"} Jul 23 18:28:21.503351 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.502975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f64b76ccb-ws6sq" event={"ID":"a11d049a-2bcc-4611-a738-068426655829","Type":"ContainerDied","Data":"f23580a731ae1d5968d38c865657732a4f772b30570766a26f1092657e5f2be4"} Jul 23 18:28:21.503351 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.502999 2576 scope.go:117] "RemoveContainer" containerID="a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b" Jul 23 18:28:21.513887 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.513866 2576 scope.go:117] "RemoveContainer" containerID="a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b" Jul 23 18:28:21.514148 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:28:21.514130 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b\": container with ID starting with a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b not found: ID does not exist" containerID="a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b" Jul 23 18:28:21.514197 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.514157 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b"} err="failed to get container status \"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b\": rpc error: code = NotFound desc = could not find container \"a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b\": container with ID starting with a70ef95b5425bda5a4a5f11b30281e1bdb9cc3ff1a06c28874e0c03f53a12f9b not found: ID does not exist" Jul 23 18:28:21.551439 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.551415 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:28:21.558197 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.558176 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f64b76ccb-ws6sq"] Jul 23 18:28:21.834504 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:28:21.834427 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11d049a-2bcc-4611-a738-068426655829" path="/var/lib/kubelet/pods/a11d049a-2bcc-4611-a738-068426655829/volumes" Jul 23 18:30:23.797987 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:30:23.797930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:30:25.287564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:30:25.287526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:03.761750 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:03.761699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:04.035980 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:04.035862 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:04.261354 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:04.261321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:09.646960 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:09.646863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:09.731898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:09.731863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:10.791156 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:10.791122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:10.860425 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:10.860382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:10.948587 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:10.948551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:11.046376 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:11.046280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:11.205776 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:11.205734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:13.757303 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:13.757276 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:31:13.759914 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:13.759887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:31:13.761164 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:13.761141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:31:13.763642 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:13.763618 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:31:16.808533 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:16.808472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:31:16.878793 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:31:16.878760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:35:39.792548 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:35:39.792411 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:35:41.386278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:35:41.386204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:36:13.790424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:36:13.790396 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:36:13.793834 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:36:13.793812 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:36:13.793974 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:36:13.793888 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:36:13.797118 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:36:13.797102 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:37:28.434938 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:37:28.434849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:37:55.612289 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:37:55.612257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:38:59.407694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.407612 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:38:59.408153 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.408136 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11d049a-2bcc-4611-a738-068426655829" containerName="authorino" Jul 23 18:38:59.408198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.408155 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11d049a-2bcc-4611-a738-068426655829" containerName="authorino" Jul 23 18:38:59.408263 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.408253 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11d049a-2bcc-4611-a738-068426655829" containerName="authorino" Jul 23 18:38:59.411417 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.411401 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.414108 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.414088 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8c025628-kserve-self-signed-certs\"" Jul 23 18:38:59.430184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.430153 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:38:59.434723 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.434700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:38:59.521392 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.521496 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.521538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pslg\" (UniqueName: \"kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.521582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.521582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521567 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.521650 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.521628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.622972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.622944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.622978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.622997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623041 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pslg\" (UniqueName: \"kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623435 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623412 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.623626 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.623536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.625327 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.625302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.625588 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.625571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.631364 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.631343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pslg\" (UniqueName: \"kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg\") pod \"e2e-same-id-a-8c025628-kserve-855dcb695-522hs\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.720538 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.720502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:38:59.853895 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:38:59.853871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:38:59.854897 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:38:59.854869 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a64f9cb_7f4b_4eb7_804d_bb7186ced05c.slice/crio-8e96bd70c1e45d4dff516043ed76d76f95279971c05690b5d2baaf6a2f0ff872 WatchSource:0}: Error finding container 8e96bd70c1e45d4dff516043ed76d76f95279971c05690b5d2baaf6a2f0ff872: Status 404 returned error can't find the container with id 8e96bd70c1e45d4dff516043ed76d76f95279971c05690b5d2baaf6a2f0ff872 Jul 23 18:39:00.863676 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:00.863643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" event={"ID":"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c","Type":"ContainerStarted","Data":"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0"} Jul 23 18:39:00.864023 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:00.863684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" event={"ID":"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c","Type":"ContainerStarted","Data":"8e96bd70c1e45d4dff516043ed76d76f95279971c05690b5d2baaf6a2f0ff872"} Jul 23 18:39:00.883270 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:00.883211 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" podStartSLOduration=1.8831968959999998 podStartE2EDuration="1.883196896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:00.881651279 +0000 UTC m=+1367.612326391" watchObservedRunningTime="2026-07-23 18:39:00.883196896 +0000 UTC m=+1367.613871987" Jul 23 18:39:09.721577 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:09.721532 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:39:09.721577 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:09.721590 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:39:09.734352 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:09.734330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:39:09.921095 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:09.921059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:39:28.735341 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:28.735299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:39:47.636140 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:47.636105 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:39:47.659705 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:47.659679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:39:48.976647 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:39:48.976619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:40:02.656758 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.656714 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" podUID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" containerName="main" containerID="cri-o://06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0" gracePeriod=45 Jul 23 18:40:02.913568 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.913509 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:40:02.955134 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955183 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pslg\" (UniqueName: \"kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955219 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955314 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home\") pod \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\" (UID: \"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c\") " Jul 23 18:40:02.955606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955443 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:02.955606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:02.955707 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955655 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home" (OuterVolumeSpecName: "home") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:02.956077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955830 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:02.956077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955853 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:02.956077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.955866 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:02.957627 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.957588 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:40:02.957627 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.957600 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg" (OuterVolumeSpecName: "kube-api-access-7pslg") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "kube-api-access-7pslg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:02.957749 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:02.957663 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm" (OuterVolumeSpecName: "dshm") pod "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" (UID: "8a64f9cb-7f4b-4eb7-804d-bb7186ced05c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:40:03.057527 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.057489 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pslg\" (UniqueName: \"kubernetes.io/projected/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-kube-api-access-7pslg\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:03.057527 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.057526 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:03.057728 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.057540 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:40:03.191962 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.191883 2576 generic.go:356] "Generic (PLEG): container finished" podID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" containerID="06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0" exitCode=0 Jul 23 18:40:03.191962 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.191923 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" Jul 23 18:40:03.192184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.191976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" event={"ID":"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c","Type":"ContainerDied","Data":"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0"} Jul 23 18:40:03.192184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.192019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs" event={"ID":"8a64f9cb-7f4b-4eb7-804d-bb7186ced05c","Type":"ContainerDied","Data":"8e96bd70c1e45d4dff516043ed76d76f95279971c05690b5d2baaf6a2f0ff872"} Jul 23 18:40:03.192184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.192043 2576 scope.go:117] "RemoveContainer" containerID="06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0" Jul 23 18:40:03.201302 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.201266 2576 scope.go:117] "RemoveContainer" containerID="06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0" Jul 23 18:40:03.201525 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:40:03.201500 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0\": container with ID starting with 06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0 not found: ID does not exist" containerID="06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0" Jul 23 18:40:03.201574 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.201533 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0"} err="failed to get container status \"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0\": rpc error: code = NotFound desc = could not find container \"06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0\": container with ID starting with 06be3344f2dbd415b4c8f8dc35957682b9af2a6d67b844fcb9a3f9a77df054d0 not found: ID does not exist" Jul 23 18:40:03.213566 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.213544 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:40:03.218316 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.218297 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8c025628-kserve-855dcb695-522hs"] Jul 23 18:40:03.843447 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:03.843407 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" path="/var/lib/kubelet/pods/8a64f9cb-7f4b-4eb7-804d-bb7186ced05c/volumes" Jul 23 18:40:11.436311 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:11.436271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:40:11.539087 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:40:11.539054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:41:13.822424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:13.822397 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:41:13.826279 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:13.826255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:41:13.827200 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:13.827180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:41:13.837878 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:13.837853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:41:43.177659 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:43.177585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:41:55.267386 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:41:55.267352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:42:03.098841 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:42:03.098805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:42:53.044533 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:42:53.044498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:14.427374 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:14.427293 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:17.531010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:17.530968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:17.631886 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:17.631853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:17.972920 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:17.972892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:23.669293 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:23.669252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:23.766127 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:23.766092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:23.889553 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:23.889519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:24.223961 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:24.223933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:24.364856 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:24.364827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:26.374954 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:26.374922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:26.485899 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:26.485873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:26.635624 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:26.635556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:26.734155 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:26.734125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:27.155303 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:27.155272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:43:27.261501 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:43:27.261474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:02.575475 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:02.575433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:07.019782 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:07.019728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:07.194008 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:07.193962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:07.466578 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:07.466549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:07.773856 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:07.773764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:07.873085 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:07.873023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:08.017413 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:08.017382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:08.502863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:08.502835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:08.825151 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:08.825048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:10.765556 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:10.765526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:10.998009 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:10.997974 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:11.396975 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:11.396939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:11.499550 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:11.499495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:46.355109 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:46.355026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:50.848658 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:50.848623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:51.014766 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:51.014733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:51.243546 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:51.243513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:51.494113 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:51.494043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:51.862384 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:51.862309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:52.083493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:52.083459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:53.889253 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:53.889195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:53.983002 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:53.982950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:54.098716 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:54.098684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:54.314163 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:54.314126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:55.335939 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:55.335904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:44:55.934737 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:44:55.934706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:30.276318 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:30.276281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:34.731438 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:34.731386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:35.934912 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:35.934873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:36.144602 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:36.144562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:36.372072 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:36.372037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:36.722950 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:36.722910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:36.977549 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:36.977446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:38.317989 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:38.317953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:39.102496 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:39.102459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:39.200263 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:39.200213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:39.273822 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:39.273792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:39.528919 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:39.528885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:39.947855 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:39.947820 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:40.050811 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:40.050776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:41.551189 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:41.551156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:41.646152 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:41.646122 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:47.856387 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:47.856348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:45:47.965946 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:45:47.965908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:00.977790 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:00.977756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:01.133879 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:01.133847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:13.862463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:13.862436 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:46:13.865860 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:13.865838 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:46:13.868541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:13.868519 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:46:13.871853 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:13.871833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:46:14.791821 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:14.791767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:19.067703 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:19.067669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:19.229400 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:19.229363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:19.474737 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:19.474706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:19.992283 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:19.992242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:20.079465 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:20.079426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:20.178210 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:20.178175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:21.628755 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:21.628720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:27.929832 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:27.929791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:28.054370 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:28.054331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:28.144588 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:28.144548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:28.244758 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:28.244727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:33.842180 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:33.842138 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:46:59.758497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:46:59.758444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:06.508962 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:06.508934 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:06.623617 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:06.623586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:06.769744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:06.769647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:22.657810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:22.657769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:28.446291 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:28.446249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:28.857057 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:28.856943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:28.963540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:28.963490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:39.181331 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:39.181248 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:39.967811 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:39.967776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:46.595282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:46.595212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:46.774917 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:46.774881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:46.870059 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:46.869970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:47.728997 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:47.728957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:47.826116 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:47.826074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:47.890834 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:47.890803 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:49.648551 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:49.648509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:55.436007 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:55.435975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:55.521522 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:55.521479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:47:56.112655 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:47:56.112616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:02.422054 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:02.422014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:02.496898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:02.496869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:02.695569 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:02.695533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:03.204998 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:03.204965 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:03.332618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:03.332574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:06.398759 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:06.398714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:06.494876 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:06.494840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:06.590456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:06.590427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:06.995429 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:06.995397 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:07.094782 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:07.094748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:07.194562 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:07.194528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:10.981022 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:10.980993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:11.664156 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:11.664121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:11.768716 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:11.768686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:11.977474 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:11.977424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:18.176380 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:18.176343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:18.274790 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:18.274763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:29.066396 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:29.066360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:33.632601 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:33.632561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:34.022850 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:34.022817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:39.829243 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:39.829196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:39.931924 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:39.931891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:40.182705 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:40.182627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:40.416966 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:40.416932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:40.524697 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:40.524660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:40.624602 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:40.624573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:40.723708 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:40.723673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:46.510265 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:46.510210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:46.612994 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:46.612962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:46.711493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:46.711454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:46.882088 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:46.882003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:52.951415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:52.951346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:53.028937 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:53.028906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:48:59.891002 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:48:59.890940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:00.216370 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:00.216333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:00.480836 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:00.480768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:03.674235 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:03.674193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:03.785045 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:03.785018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:03.866606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:03.866571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:04.495801 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:04.495762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:04.579198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:04.579169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:08.215947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:08.215858 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:15.320970 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:15.320938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:16.087288 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:16.087252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:19.702296 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.702261 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:49:19.702677 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.702633 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" containerName="main" Jul 23 18:49:19.702677 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.702646 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" containerName="main" Jul 23 18:49:19.702749 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.702719 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a64f9cb-7f4b-4eb7-804d-bb7186ced05c" containerName="main" Jul 23 18:49:19.705722 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.705705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.708557 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.708538 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d6a5d58\"/\"default-dockercfg-szzgg\"" Jul 23 18:49:19.709459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.709438 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d6a5d58\"/\"openshift-service-ca.crt\"" Jul 23 18:49:19.709459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.709449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d6a5d58\"/\"e2e-lifecycle-model-3d6a5d58-kserve-self-signed-certs\"" Jul 23 18:49:19.709636 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.709498 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d6a5d58\"/\"kube-root-ca.crt\"" Jul 23 18:49:19.720950 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.720932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:49:19.731645 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.731619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:49:19.877629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.877810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.877810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.877810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfq4\" (UniqueName: \"kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.877810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.878043 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.877835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978685 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978685 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978877 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978877 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfq4\" (UniqueName: \"kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978877 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.978877 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.978784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.979142 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.979118 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.979200 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.979136 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.979273 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.979209 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.980810 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.980785 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.981244 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.981206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:19.990083 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:19.990059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfq4\" (UniqueName: \"kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4\") pod \"e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:20.015024 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:20.014999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:20.156796 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:20.156774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:49:20.159569 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:49:20.159534 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b78044_b27b_4143_8cf1_5bd4e87585ca.slice/crio-e5a543129aa9aa35313a7c94e67c1e0a3a4aec74ec8c6a7bc01876efd7128179 WatchSource:0}: Error finding container e5a543129aa9aa35313a7c94e67c1e0a3a4aec74ec8c6a7bc01876efd7128179: Status 404 returned error can't find the container with id e5a543129aa9aa35313a7c94e67c1e0a3a4aec74ec8c6a7bc01876efd7128179 Jul 23 18:49:21.113269 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:21.113213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" event={"ID":"d4b78044-b27b-4143-8cf1-5bd4e87585ca","Type":"ContainerStarted","Data":"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf"} Jul 23 18:49:21.113644 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:21.113275 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" event={"ID":"d4b78044-b27b-4143-8cf1-5bd4e87585ca","Type":"ContainerStarted","Data":"e5a543129aa9aa35313a7c94e67c1e0a3a4aec74ec8c6a7bc01876efd7128179"} Jul 23 18:49:21.132433 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:21.132385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" podStartSLOduration=2.132370073 podStartE2EDuration="2.132370073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:21.129950542 +0000 UTC m=+1987.860625660" watchObservedRunningTime="2026-07-23 18:49:21.132370073 +0000 UTC m=+1987.863045163" Jul 23 18:49:30.015465 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:30.015431 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:30.015465 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:30.015472 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:30.027772 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:30.027743 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:49:30.174069 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:49:30.174043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:50:02.341601 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:02.341556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:06.812132 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:06.812095 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:09.051421 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:09.051383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:14.042062 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:14.042026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:50:14.127844 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:14.127812 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:29.062241 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.062151 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" podUID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" containerName="main" containerID="cri-o://4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf" gracePeriod=45 Jul 23 18:50:29.311487 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.311466 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:50:29.340450 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340390 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340450 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340429 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340629 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340511 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340735 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340652 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:50:29.340735 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340703 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340734 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:50:29.340826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340744 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkfq4\" (UniqueName: \"kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4\") pod \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\" (UID: \"d4b78044-b27b-4143-8cf1-5bd4e87585ca\") " Jul 23 18:50:29.340826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.340747 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home" (OuterVolumeSpecName: "home") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:50:29.341101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.341082 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.341101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.341103 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.341294 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.341114 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.343239 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.343203 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm" (OuterVolumeSpecName: "dshm") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:50:29.343339 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.343282 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:50:29.343402 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.343344 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4" (OuterVolumeSpecName: "kube-api-access-kkfq4") pod "d4b78044-b27b-4143-8cf1-5bd4e87585ca" (UID: "d4b78044-b27b-4143-8cf1-5bd4e87585ca"). InnerVolumeSpecName "kube-api-access-kkfq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:50:29.441456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.441431 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b78044-b27b-4143-8cf1-5bd4e87585ca-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.441456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.441453 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b78044-b27b-4143-8cf1-5bd4e87585ca-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.441611 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.441462 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfq4\" (UniqueName: \"kubernetes.io/projected/d4b78044-b27b-4143-8cf1-5bd4e87585ca-kube-api-access-kkfq4\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:50:29.482628 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.482594 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" containerID="4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf" exitCode=0 Jul 23 18:50:29.482722 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.482633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" event={"ID":"d4b78044-b27b-4143-8cf1-5bd4e87585ca","Type":"ContainerDied","Data":"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf"} Jul 23 18:50:29.482722 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.482636 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" Jul 23 18:50:29.482722 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.482662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k" event={"ID":"d4b78044-b27b-4143-8cf1-5bd4e87585ca","Type":"ContainerDied","Data":"e5a543129aa9aa35313a7c94e67c1e0a3a4aec74ec8c6a7bc01876efd7128179"} Jul 23 18:50:29.482722 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.482684 2576 scope.go:117] "RemoveContainer" containerID="4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf" Jul 23 18:50:29.494077 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.494052 2576 scope.go:117] "RemoveContainer" containerID="4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf" Jul 23 18:50:29.494441 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:50:29.494420 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf\": container with ID starting with 4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf not found: ID does not exist" containerID="4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf" Jul 23 18:50:29.494532 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.494448 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf"} err="failed to get container status \"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf\": rpc error: code = NotFound desc = could not find container \"4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf\": container with ID starting with 4c996650483e2dfa597f7e258114c37a38d06e8711767a37b9565f950902cfbf not found: ID does not exist" Jul 23 18:50:29.504794 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.504775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:50:29.509454 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.509435 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d6a5d58/e2e-lifecycle-model-3d6a5d58-kserve-76549b7879-4ss4k"] Jul 23 18:50:29.834471 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:29.834446 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" path="/var/lib/kubelet/pods/d4b78044-b27b-4143-8cf1-5bd4e87585ca/volumes" Jul 23 18:50:48.701481 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:48.701359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:48.800980 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:48.800948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:51.402416 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:51.402363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:51.592564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:51.592510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:50:57.292053 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:50:57.292011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:03.280040 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:03.280004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:03.381093 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:03.381024 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:03.581846 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:03.581750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:03.827730 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:03.827696 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:04.944742 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:04.944713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:05.033719 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:05.033688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:05.133742 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:05.133708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:05.900317 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:05.900266 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:05.995316 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:05.995277 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:06.098436 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:06.098409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:07.505733 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:07.505692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:13.651986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.651950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:13.755811 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.755767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:13.897537 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.897512 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:51:13.901775 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.901755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:51:13.906354 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.906302 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:51:13.909686 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:13.909670 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:51:14.336381 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:14.336350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:21.091987 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:21.091939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:21.203046 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:21.203000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:21.298390 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:21.298354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:31.824753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:31.824692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:32.250493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:32.250465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:46.327971 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:46.327931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:46.384888 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:46.384856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:53.170101 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:53.170059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:59.428000 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:59.427966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:59.529044 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:59.529011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:51:59.775889 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:51:59.775859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:00.234691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:00.234629 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:00.567200 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:00.567120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:03.573132 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:03.573097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:05.204081 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:05.204006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:08.074348 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:08.074316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:08.452451 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:08.452415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:08.781426 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:08.781339 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:08.966495 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:08.966459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:11.643047 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:11.643014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:11.746135 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:11.746097 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:11.930065 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:11.929981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:12.487812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:12.487782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:12.590798 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:12.590768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:12.689384 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:12.689350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:16.035668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:16.035624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:25.064527 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:25.064494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:25.164617 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:25.164583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:33.046378 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:33.046332 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:38.302836 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.302799 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:52:38.303335 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.303314 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" containerName="main" Jul 23 18:52:38.303409 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.303337 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" containerName="main" Jul 23 18:52:38.303452 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.303426 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b78044-b27b-4143-8cf1-5bd4e87585ca" containerName="main" Jul 23 18:52:38.306606 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.306585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.311150 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.311131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e16158\"/\"default-dockercfg-r8nxd\"" Jul 23 18:52:38.311706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.311693 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e16158\"/\"kube-root-ca.crt\"" Jul 23 18:52:38.311782 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.311726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e16158\"/\"test-model-e16158-kserve-self-signed-certs\"" Jul 23 18:52:38.311782 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.311765 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e16158\"/\"openshift-service-ca.crt\"" Jul 23 18:52:38.320670 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.320648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:52:38.347167 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.347139 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:38.390918 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.390887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.391055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.390930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.391055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.390967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.391055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.390988 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25wlb\" (UniqueName: \"kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.391055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.391006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.391055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.391045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.491922 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.491882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.491933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.491970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.491991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25wlb\" (UniqueName: \"kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.492019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492097 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.492067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492410 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.492387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492492 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.492469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.492547 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.492496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.494266 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.494240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.494520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.494506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.501739 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.501717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25wlb\" (UniqueName: \"kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb\") pod \"test-model-e16158-kserve-76748d6985-glhf2\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.616063 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.615977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:38.750581 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:38.750553 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:52:38.755024 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:52:38.752487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44179941_d46c_4b6e_ba5f_29d6163631a9.slice/crio-4578db2122dd314205ef40ac6b7123bd26f0103dfa5b3e052c0fb7e748ffa72a WatchSource:0}: Error finding container 4578db2122dd314205ef40ac6b7123bd26f0103dfa5b3e052c0fb7e748ffa72a: Status 404 returned error can't find the container with id 4578db2122dd314205ef40ac6b7123bd26f0103dfa5b3e052c0fb7e748ffa72a Jul 23 18:52:39.174278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:39.174244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" event={"ID":"44179941-d46c-4b6e-ba5f-29d6163631a9","Type":"ContainerStarted","Data":"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857"} Jul 23 18:52:39.174278 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:39.174284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" event={"ID":"44179941-d46c-4b6e-ba5f-29d6163631a9","Type":"ContainerStarted","Data":"4578db2122dd314205ef40ac6b7123bd26f0103dfa5b3e052c0fb7e748ffa72a"} Jul 23 18:52:39.192511 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:39.192463 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" podStartSLOduration=1.192449455 podStartE2EDuration="1.192449455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:52:39.191383686 +0000 UTC m=+2185.922058778" watchObservedRunningTime="2026-07-23 18:52:39.192449455 +0000 UTC m=+2185.923124560" Jul 23 18:52:40.207581 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:40.207548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:40.309687 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:40.309655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:52:48.616669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:48.616615 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:48.616669 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:48.616681 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:48.631093 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:48.631061 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:52:49.237166 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:52:49.237136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:53:04.199127 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.199089 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:53:04.202375 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.202358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.205582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.205557 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aa7092\"/\"openshift-service-ca.crt\"" Jul 23 18:53:04.205582 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.205575 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aa7092\"/\"kube-root-ca.crt\"" Jul 23 18:53:04.206694 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.206672 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aa7092\"/\"test-model-aa7092-kserve-self-signed-certs\"" Jul 23 18:53:04.206846 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.206725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aa7092\"/\"default-dockercfg-z4r28\"" Jul 23 18:53:04.213870 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.213848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:53:04.300098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.300058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:53:04.331404 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.331520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.331520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzv2\" (UniqueName: \"kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.331520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.331642 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.331642 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.331557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.431997 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.431963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432157 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432157 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432082 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432257 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432306 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzv2\" (UniqueName: \"kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432550 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432616 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.432616 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.432593 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.434470 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.434446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.434838 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.434817 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.441513 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.441484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzv2\" (UniqueName: \"kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2\") pod \"test-model-aa7092-kserve-65c9d9c577-8kgk8\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.513154 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.513085 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:04.648273 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:04.648249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:53:04.649791 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:53:04.649763 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod449a9ba3_d2b9_45c0_8941_a4d69d555b19.slice/crio-20c8ba90ccd5ab1b35fed8a86a0071259a393474283a633e4d0df48746cd9be4 WatchSource:0}: Error finding container 20c8ba90ccd5ab1b35fed8a86a0071259a393474283a633e4d0df48746cd9be4: Status 404 returned error can't find the container with id 20c8ba90ccd5ab1b35fed8a86a0071259a393474283a633e4d0df48746cd9be4 Jul 23 18:53:05.309066 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:05.309028 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" event={"ID":"449a9ba3-d2b9-45c0-8941-a4d69d555b19","Type":"ContainerStarted","Data":"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3"} Jul 23 18:53:05.309571 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:05.309084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" event={"ID":"449a9ba3-d2b9-45c0-8941-a4d69d555b19","Type":"ContainerStarted","Data":"20c8ba90ccd5ab1b35fed8a86a0071259a393474283a633e4d0df48746cd9be4"} Jul 23 18:53:05.331144 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:05.331080 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" podStartSLOduration=1.331066156 podStartE2EDuration="1.331066156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:53:05.329758531 +0000 UTC m=+2212.060433615" watchObservedRunningTime="2026-07-23 18:53:05.331066156 +0000 UTC m=+2212.061741246" Jul 23 18:53:14.513509 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:14.513459 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:14.513509 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:14.513522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:14.526014 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:14.525992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:15.371026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:15.370997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:53:46.019075 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:46.018988 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:53:46.029490 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:46.029458 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:53:46.898667 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:46.898632 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:53:46.916514 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:46.916487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:53:47.662357 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:47.662325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:53:47.766945 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:47.766916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:53:57.568941 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:53:57.568905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:01.040852 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.040794 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" podUID="44179941-d46c-4b6e-ba5f-29d6163631a9" containerName="main" containerID="cri-o://c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857" gracePeriod=45 Jul 23 18:54:01.291877 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.291821 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:54:01.424932 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.424884 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.424932 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.424941 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.425203 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.424969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.425203 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425007 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.425203 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.425203 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425133 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25wlb\" (UniqueName: \"kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb\") pod \"44179941-d46c-4b6e-ba5f-29d6163631a9\" (UID: \"44179941-d46c-4b6e-ba5f-29d6163631a9\") " Jul 23 18:54:01.425424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:01.425424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425254 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:01.425424 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425371 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home" (OuterVolumeSpecName: "home") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:01.425544 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425459 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.425544 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425474 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.425544 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.425494 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.427282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.427255 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm" (OuterVolumeSpecName: "dshm") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:01.427282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.427264 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:54:01.427436 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.427312 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb" (OuterVolumeSpecName: "kube-api-access-25wlb") pod "44179941-d46c-4b6e-ba5f-29d6163631a9" (UID: "44179941-d46c-4b6e-ba5f-29d6163631a9"). InnerVolumeSpecName "kube-api-access-25wlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:54:01.526041 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.526017 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44179941-d46c-4b6e-ba5f-29d6163631a9-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.526041 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.526037 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44179941-d46c-4b6e-ba5f-29d6163631a9-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.526191 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.526046 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25wlb\" (UniqueName: \"kubernetes.io/projected/44179941-d46c-4b6e-ba5f-29d6163631a9-kube-api-access-25wlb\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:01.604741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.604659 2576 generic.go:356] "Generic (PLEG): container finished" podID="44179941-d46c-4b6e-ba5f-29d6163631a9" containerID="c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857" exitCode=0 Jul 23 18:54:01.604741 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.604705 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" Jul 23 18:54:01.604909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.604742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" event={"ID":"44179941-d46c-4b6e-ba5f-29d6163631a9","Type":"ContainerDied","Data":"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857"} Jul 23 18:54:01.604909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.604787 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2" event={"ID":"44179941-d46c-4b6e-ba5f-29d6163631a9","Type":"ContainerDied","Data":"4578db2122dd314205ef40ac6b7123bd26f0103dfa5b3e052c0fb7e748ffa72a"} Jul 23 18:54:01.604909 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.604807 2576 scope.go:117] "RemoveContainer" containerID="c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857" Jul 23 18:54:01.613897 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.613877 2576 scope.go:117] "RemoveContainer" containerID="c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857" Jul 23 18:54:01.614127 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:54:01.614109 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857\": container with ID starting with c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857 not found: ID does not exist" containerID="c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857" Jul 23 18:54:01.614175 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.614134 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857"} err="failed to get container status \"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857\": rpc error: code = NotFound desc = could not find container \"c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857\": container with ID starting with c358b4f30292f656d042b41e6f9bf67703f5b08b638fc55614e750355194b857 not found: ID does not exist" Jul 23 18:54:01.629691 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.629665 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:54:01.637864 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.637841 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e16158/test-model-e16158-kserve-76748d6985-glhf2"] Jul 23 18:54:01.836033 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.835993 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44179941-d46c-4b6e-ba5f-29d6163631a9" path="/var/lib/kubelet/pods/44179941-d46c-4b6e-ba5f-29d6163631a9/volumes" Jul 23 18:54:01.919098 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:01.919008 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" podUID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" containerName="main" containerID="cri-o://303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3" gracePeriod=45 Jul 23 18:54:02.144744 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.144702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:02.169426 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.169371 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:54:02.232337 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232305 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232467 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232342 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232467 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvzv2\" (UniqueName: \"kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232467 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232432 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232551 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232564 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home" (OuterVolumeSpecName: "home") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:02.232632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache\") pod \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\" (UID: \"449a9ba3-d2b9-45c0-8941-a4d69d555b19\") " Jul 23 18:54:02.232774 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232659 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:02.232829 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232789 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache" (OuterVolumeSpecName: "model-cache") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:02.233003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.232987 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.233003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.233004 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.233003 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.233013 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.234698 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.234680 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm" (OuterVolumeSpecName: "dshm") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:54:02.234968 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.234949 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2" (OuterVolumeSpecName: "kube-api-access-pvzv2") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "kube-api-access-pvzv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:54:02.235032 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.234953 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "449a9ba3-d2b9-45c0-8941-a4d69d555b19" (UID: "449a9ba3-d2b9-45c0-8941-a4d69d555b19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:54:02.333807 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.333785 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/449a9ba3-d2b9-45c0-8941-a4d69d555b19-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.333807 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.333805 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/449a9ba3-d2b9-45c0-8941-a4d69d555b19-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.333938 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.333816 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzv2\" (UniqueName: \"kubernetes.io/projected/449a9ba3-d2b9-45c0-8941-a4d69d555b19-kube-api-access-pvzv2\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:54:02.535456 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.535425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:02.612343 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.612317 2576 generic.go:356] "Generic (PLEG): container finished" podID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" containerID="303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3" exitCode=0 Jul 23 18:54:02.612478 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.612357 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" Jul 23 18:54:02.612478 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.612400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" event={"ID":"449a9ba3-d2b9-45c0-8941-a4d69d555b19","Type":"ContainerDied","Data":"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3"} Jul 23 18:54:02.612478 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.612442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8" event={"ID":"449a9ba3-d2b9-45c0-8941-a4d69d555b19","Type":"ContainerDied","Data":"20c8ba90ccd5ab1b35fed8a86a0071259a393474283a633e4d0df48746cd9be4"} Jul 23 18:54:02.612478 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.612458 2576 scope.go:117] "RemoveContainer" containerID="303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3" Jul 23 18:54:02.623524 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.623508 2576 scope.go:117] "RemoveContainer" containerID="303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3" Jul 23 18:54:02.623778 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:54:02.623756 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3\": container with ID starting with 303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3 not found: ID does not exist" containerID="303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3" Jul 23 18:54:02.623852 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.623786 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3"} err="failed to get container status \"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3\": rpc error: code = NotFound desc = could not find container \"303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3\": container with ID starting with 303627ff360fdee002780f5ffba3dd6966fad34db5981737a11192736c8dadf3 not found: ID does not exist" Jul 23 18:54:02.636904 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.636879 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:54:02.644026 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:02.644006 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-aa7092/test-model-aa7092-kserve-65c9d9c577-8kgk8"] Jul 23 18:54:03.834490 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:03.834461 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" path="/var/lib/kubelet/pods/449a9ba3-d2b9-45c0-8941-a4d69d555b19/volumes" Jul 23 18:54:08.638812 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:08.638779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:08.712318 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:08.712274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:08.947092 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:08.947066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:09.180972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:09.180944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:09.808259 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:09.808194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:09.907367 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:09.907337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:19.733212 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:19.733165 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:24.391459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:24.391422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:24.738271 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:24.738216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:30.719436 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:30.719401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:30.814268 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:30.814203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:30.979002 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:30.978901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:31.212397 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:31.212361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:31.532390 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:31.532359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:31.859799 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:31.859707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:31.958912 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:31.958879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:35.372923 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:35.372890 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:35.478055 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:35.478029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:35.579463 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:35.579426 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:35.890378 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:35.890345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:36.092494 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:36.092462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:36.194203 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:36.194092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:39.761835 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:39.761773 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:40.292737 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:40.292687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:40.593751 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:40.593655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:40.692901 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:40.692867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:43.288279 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:43.288246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:43.401820 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:43.401794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:43.475978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:43.475953 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:43.821833 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:43.821785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:44.028879 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:44.028852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:44.131631 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:44.131560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:47.760541 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:47.760488 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:56.473370 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:56.473335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:54:56.571550 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:54:56.571506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:04.703883 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:04.703844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:05.062441 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:05.062353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:05.161286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:05.161251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:06.322195 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:06.322163 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:06.426663 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:06.426631 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:11.608530 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:11.608497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:15.900304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:15.900267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:16.256655 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:16.256624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:22.242361 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:22.242321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:22.309460 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:22.309427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:22.387885 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:22.387853 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:22.601825 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:22.601737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:23.362956 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:23.362916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:23.463411 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:23.463383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:29.027121 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:29.027079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:38.369100 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:38.369063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:42.803976 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:42.803939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:43.201147 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:43.201119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:49.212293 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:49.212249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:49.411073 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:49.411042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:49.495759 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:49.495681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:49.662947 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:49.662902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:49.763287 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:49.763184 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:50.046197 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:50.046110 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:50.373603 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:50.373506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:50.663682 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:50.663593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:53.652143 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:53.652108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:53.738110 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:53.738078 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:53.811411 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:53.811377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:54.076824 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:54.076791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:54.344668 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:54.344590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:54.445786 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:54.445752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:58.168415 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:58.168383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:58.524803 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:58.524769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:58.988555 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:58.988517 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:55:59.193599 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:55:59.193570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:01.859624 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:01.859587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:01.971503 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:01.971470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:02.174083 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:02.173987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:02.477313 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:02.477279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:02.919046 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:02.918968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:03.033599 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:03.033563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:06.260469 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:06.260429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:13.931628 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:13.931597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:56:13.935091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:13.935068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:56:13.939891 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:13.939875 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 18:56:13.943574 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:13.943556 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 18:56:15.142826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:15.142799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:15.244942 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:15.244913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:23.844643 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:23.844594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:23.949756 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:23.949722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:28.737542 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737507 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:56:28.737898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737831 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44179941-d46c-4b6e-ba5f-29d6163631a9" containerName="main" Jul 23 18:56:28.737898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737842 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44179941-d46c-4b6e-ba5f-29d6163631a9" containerName="main" Jul 23 18:56:28.737898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737861 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" containerName="main" Jul 23 18:56:28.737898 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737866 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" containerName="main" Jul 23 18:56:28.738073 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737923 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44179941-d46c-4b6e-ba5f-29d6163631a9" containerName="main" Jul 23 18:56:28.738073 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.737934 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="449a9ba3-d2b9-45c0-8941-a4d69d555b19" containerName="main" Jul 23 18:56:28.741033 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.741015 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.744734 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.744711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"auth-test-model-3f12e9-kserve-self-signed-certs\"" Jul 23 18:56:28.744884 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.744783 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"default-dockercfg-56pzq\"" Jul 23 18:56:28.744884 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.744723 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"kube-root-ca.crt\"" Jul 23 18:56:28.744884 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.744791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f12e9\"/\"openshift-service-ca.crt\"" Jul 23 18:56:28.753737 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.753709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:56:28.782828 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.782801 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:28.905385 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.905497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqmp\" (UniqueName: \"kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.905497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.905497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.905497 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905481 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:28.905641 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:28.905546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqmp\" (UniqueName: \"kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006184 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006457 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006457 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006457 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006624 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006624 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.006703 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.006667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.008461 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.008442 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.008753 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.008734 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.014099 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.014078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqmp\" (UniqueName: \"kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp\") pod \"auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.052065 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.052045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:29.185493 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.185468 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:56:29.186951 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:56:29.186927 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0a2a759_9799_4594_820b_5b590b69992a.slice/crio-97f863918bbf4342edc655820b3c08b5faa2399bec1cd68a2372c4672937da67 WatchSource:0}: Error finding container 97f863918bbf4342edc655820b3c08b5faa2399bec1cd68a2372c4672937da67: Status 404 returned error can't find the container with id 97f863918bbf4342edc655820b3c08b5faa2399bec1cd68a2372c4672937da67 Jul 23 18:56:29.370529 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.370455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" event={"ID":"d0a2a759-9799-4594-820b-5b590b69992a","Type":"ContainerStarted","Data":"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b"} Jul 23 18:56:29.370529 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.370494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" event={"ID":"d0a2a759-9799-4594-820b-5b590b69992a","Type":"ContainerStarted","Data":"97f863918bbf4342edc655820b3c08b5faa2399bec1cd68a2372c4672937da67"} Jul 23 18:56:29.390570 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:29.390519 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" podStartSLOduration=1.390503023 podStartE2EDuration="1.390503023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:56:29.388465178 +0000 UTC m=+2416.119140270" watchObservedRunningTime="2026-07-23 18:56:29.390503023 +0000 UTC m=+2416.121178113" Jul 23 18:56:37.197286 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:37.197190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:37.260319 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:37.260298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:56:39.052643 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:39.052607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:39.053148 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:39.052912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:39.065594 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:39.065571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:56:39.434653 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:56:39.434576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:57:00.810295 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.810257 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:57:00.814019 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.813997 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.816239 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.816202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"kube-root-ca.crt\"" Jul 23 18:57:00.817150 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.817128 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"default-dockercfg-2lczk\"" Jul 23 18:57:00.817298 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.817152 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"openshift-service-ca.crt\"" Jul 23 18:57:00.817298 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.817131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"auth-test-model-60de5a-kserve-self-signed-certs\"" Jul 23 18:57:00.824198 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.824176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:57:00.846445 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.846562 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.846562 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.846562 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.846706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.846706 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.846622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd887\" (UniqueName: \"kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.852405 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.852381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:57:00.947889 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.947864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.947995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.947892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd887\" (UniqueName: \"kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.947995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.947950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.947995 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.947970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.948168 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.948046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.948168 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.948090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.948632 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.948611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.948739 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.948689 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.948836 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.948768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.950864 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.950841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.951261 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.951241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:00.956552 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:00.956528 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd887\" (UniqueName: \"kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887\") pod \"auth-test-model-60de5a-kserve-66d8ff98c9-wssj7\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:01.124491 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:01.124415 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:01.258200 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:01.258171 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:57:01.259913 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:57:01.259892 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc3b63b_0b35_495e_8507_9f0f4ced7e49.slice/crio-65cada1302c007da63123b54aef37e08df26192038713996da64be9e6d319325 WatchSource:0}: Error finding container 65cada1302c007da63123b54aef37e08df26192038713996da64be9e6d319325: Status 404 returned error can't find the container with id 65cada1302c007da63123b54aef37e08df26192038713996da64be9e6d319325 Jul 23 18:57:01.542826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:01.542793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" event={"ID":"6bc3b63b-0b35-495e-8507-9f0f4ced7e49","Type":"ContainerStarted","Data":"9bafc58a4c87b65837fd1a306da53b4533737b050f16f9527145cc840aaea369"} Jul 23 18:57:01.542826 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:01.542831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" event={"ID":"6bc3b63b-0b35-495e-8507-9f0f4ced7e49","Type":"ContainerStarted","Data":"65cada1302c007da63123b54aef37e08df26192038713996da64be9e6d319325"} Jul 23 18:57:01.561934 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:01.561881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" podStartSLOduration=1.561863064 podStartE2EDuration="1.561863064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:57:01.561213812 +0000 UTC m=+2448.291888924" watchObservedRunningTime="2026-07-23 18:57:01.561863064 +0000 UTC m=+2448.292538157" Jul 23 18:57:11.125540 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:11.125491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:11.126016 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:11.125584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:11.138486 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:11.138460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:57:11.609070 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:57:11.609037 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:58:08.933852 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:08.933756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:58:11.136447 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:11.136372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:58:31.082698 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:31.082665 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:58:31.143978 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:31.143951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:58:59.276451 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:59.276419 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:58:59.294818 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:58:59.294782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:59:00.033030 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:00.033000 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:59:00.046618 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:00.046568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:59:13.639581 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:13.639542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:59:14.296557 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.296495 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" podUID="d0a2a759-9799-4594-820b-5b590b69992a" containerName="main" containerID="cri-o://68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b" gracePeriod=45 Jul 23 18:59:14.550972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.550907 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:59:14.712389 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712408 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712495 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712518 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsqmp\" (UniqueName: \"kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp\") pod \"d0a2a759-9799-4594-820b-5b590b69992a\" (UID: \"d0a2a759-9799-4594-820b-5b590b69992a\") " Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:14.712863 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712755 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:14.713332 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.712896 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home" (OuterVolumeSpecName: "home") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:14.714809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.714787 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm" (OuterVolumeSpecName: "dshm") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:14.714809 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.714794 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:59:14.715031 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.715004 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp" (OuterVolumeSpecName: "kube-api-access-gsqmp") pod "d0a2a759-9799-4594-820b-5b590b69992a" (UID: "d0a2a759-9799-4594-820b-5b590b69992a"). InnerVolumeSpecName "kube-api-access-gsqmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814020 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814046 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0a2a759-9799-4594-820b-5b590b69992a-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814057 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814066 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqmp\" (UniqueName: \"kubernetes.io/projected/d0a2a759-9799-4594-820b-5b590b69992a-kube-api-access-gsqmp\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814074 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:14.814090 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:14.814083 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0a2a759-9799-4594-820b-5b590b69992a-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.053083 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.053039 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" podUID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" containerName="main" containerID="cri-o://9bafc58a4c87b65837fd1a306da53b4533737b050f16f9527145cc840aaea369" gracePeriod=45 Jul 23 18:59:15.246406 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.246373 2576 generic.go:356] "Generic (PLEG): container finished" podID="d0a2a759-9799-4594-820b-5b590b69992a" containerID="68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b" exitCode=0 Jul 23 18:59:15.246564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.246428 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" Jul 23 18:59:15.246564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.246446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" event={"ID":"d0a2a759-9799-4594-820b-5b590b69992a","Type":"ContainerDied","Data":"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b"} Jul 23 18:59:15.246564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.246487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc" event={"ID":"d0a2a759-9799-4594-820b-5b590b69992a","Type":"ContainerDied","Data":"97f863918bbf4342edc655820b3c08b5faa2399bec1cd68a2372c4672937da67"} Jul 23 18:59:15.246564 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.246508 2576 scope.go:117] "RemoveContainer" containerID="68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b" Jul 23 18:59:15.248500 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.248473 2576 generic.go:356] "Generic (PLEG): container finished" podID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" containerID="9bafc58a4c87b65837fd1a306da53b4533737b050f16f9527145cc840aaea369" exitCode=0 Jul 23 18:59:15.248619 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.248565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" event={"ID":"6bc3b63b-0b35-495e-8507-9f0f4ced7e49","Type":"ContainerDied","Data":"9bafc58a4c87b65837fd1a306da53b4533737b050f16f9527145cc840aaea369"} Jul 23 18:59:15.256831 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.256810 2576 scope.go:117] "RemoveContainer" containerID="68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b" Jul 23 18:59:15.257145 ip-10-0-135-127 kubenswrapper[2576]: E0723 18:59:15.257110 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b\": container with ID starting with 68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b not found: ID does not exist" containerID="68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b" Jul 23 18:59:15.257343 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.257152 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b"} err="failed to get container status \"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b\": rpc error: code = NotFound desc = could not find container \"68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b\": container with ID starting with 68a608222ebe4c5dce41837c79569cd648076c9b4d3683d72975d17b0d27085b not found: ID does not exist" Jul 23 18:59:15.272520 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.272495 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:59:15.285649 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.285624 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f12e9/auth-test-model-3f12e9-kserve-cbf8cf77f-5zgnc"] Jul 23 18:59:15.298855 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.298834 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:59:15.420726 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.420726 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420702 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd887\" (UniqueName: \"kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.420726 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.420972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.420972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420797 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.420972 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.420858 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home\") pod \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\" (UID: \"6bc3b63b-0b35-495e-8507-9f0f4ced7e49\") " Jul 23 18:59:15.421145 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.421008 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:15.421145 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.421024 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:15.421282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.421170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home" (OuterVolumeSpecName: "home") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:15.421282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.421211 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.421282 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.421257 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.422951 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.422928 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm" (OuterVolumeSpecName: "dshm") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:59:15.423027 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.423013 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:59:15.423072 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.423055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887" (OuterVolumeSpecName: "kube-api-access-wd887") pod "6bc3b63b-0b35-495e-8507-9f0f4ced7e49" (UID: "6bc3b63b-0b35-495e-8507-9f0f4ced7e49"). InnerVolumeSpecName "kube-api-access-wd887". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:59:15.521844 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.521823 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.521844 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.521843 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.521974 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.521852 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd887\" (UniqueName: \"kubernetes.io/projected/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-kube-api-access-wd887\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.521974 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.521862 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc3b63b-0b35-495e-8507-9f0f4ced7e49-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 18:59:15.835182 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:15.835149 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a2a759-9799-4594-820b-5b590b69992a" path="/var/lib/kubelet/pods/d0a2a759-9799-4594-820b-5b590b69992a/volumes" Jul 23 18:59:16.256292 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:16.256261 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" Jul 23 18:59:16.256471 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:16.256264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7" event={"ID":"6bc3b63b-0b35-495e-8507-9f0f4ced7e49","Type":"ContainerDied","Data":"65cada1302c007da63123b54aef37e08df26192038713996da64be9e6d319325"} Jul 23 18:59:16.256471 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:16.256386 2576 scope.go:117] "RemoveContainer" containerID="9bafc58a4c87b65837fd1a306da53b4533737b050f16f9527145cc840aaea369" Jul 23 18:59:16.274696 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:16.274673 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:59:16.278524 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:16.278500 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/auth-test-model-60de5a-kserve-66d8ff98c9-wssj7"] Jul 23 18:59:17.835309 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:17.835270 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" path="/var/lib/kubelet/pods/6bc3b63b-0b35-495e-8507-9f0f4ced7e49/volumes" Jul 23 18:59:42.700653 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.700574 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.700921 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" containerName="main" Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.700936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" containerName="main" Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.700954 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a2a759-9799-4594-820b-5b590b69992a" containerName="main" Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.700959 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a2a759-9799-4594-820b-5b590b69992a" containerName="main" Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.701027 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a2a759-9799-4594-820b-5b590b69992a" containerName="main" Jul 23 18:59:42.701251 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.701038 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc3b63b-0b35-495e-8507-9f0f4ced7e49" containerName="main" Jul 23 18:59:42.703959 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.703935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.707158 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.707135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"default-dockercfg-2lczk\"" Jul 23 18:59:42.707304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.707168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"sub-test-model-60de5a-kserve-self-signed-certs\"" Jul 23 18:59:42.707304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.707183 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"kube-root-ca.crt\"" Jul 23 18:59:42.707304 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.707282 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-60de5a\"/\"openshift-service-ca.crt\"" Jul 23 18:59:42.714545 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.714524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 18:59:42.738010 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.737986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 18:59:42.845091 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.845206 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.845206 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.845206 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.845526 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.845600 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.845551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blbm\" (UniqueName: \"kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.945866 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.945986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.945986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.945986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.945986 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blbm\" (UniqueName: \"kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.946201 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.945993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.946309 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.946287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.946438 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.946421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.946570 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.946551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.948330 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.948311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.948484 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.948466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:42.954656 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:42.954609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blbm\" (UniqueName: \"kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm\") pod \"sub-test-model-60de5a-kserve-57488bb64f-flgk7\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:43.015519 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:43.015495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:43.149579 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:43.149558 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 18:59:43.151843 ip-10-0-135-127 kubenswrapper[2576]: W0723 18:59:43.151816 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podced595f7_cae1_477b_a1d8_85efedb07e24.slice/crio-db0df2c3330dd03a444e972e204142290f1e71ca3df2f19cd7e266108045c805 WatchSource:0}: Error finding container db0df2c3330dd03a444e972e204142290f1e71ca3df2f19cd7e266108045c805: Status 404 returned error can't find the container with id db0df2c3330dd03a444e972e204142290f1e71ca3df2f19cd7e266108045c805 Jul 23 18:59:43.402776 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:43.402696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" event={"ID":"ced595f7-cae1-477b-a1d8-85efedb07e24","Type":"ContainerStarted","Data":"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd"} Jul 23 18:59:43.402776 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:43.402738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" event={"ID":"ced595f7-cae1-477b-a1d8-85efedb07e24","Type":"ContainerStarted","Data":"db0df2c3330dd03a444e972e204142290f1e71ca3df2f19cd7e266108045c805"} Jul 23 18:59:43.422870 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:43.422822 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" podStartSLOduration=1.4228070050000001 podStartE2EDuration="1.422807005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:59:43.420563222 +0000 UTC m=+2610.151238307" watchObservedRunningTime="2026-07-23 18:59:43.422807005 +0000 UTC m=+2610.153482096" Jul 23 18:59:53.016054 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:53.016020 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:53.016459 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:53.016074 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:53.028739 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:53.028714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 18:59:53.466088 ip-10-0-135-127 kubenswrapper[2576]: I0723 18:59:53.466056 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 19:00:29.381752 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:29.381707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:00:30.125315 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:30.125280 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 19:00:30.143236 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:30.143208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:00:33.732180 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:33.732150 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:00:45.147312 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.147269 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" podUID="ced595f7-cae1-477b-a1d8-85efedb07e24" containerName="main" containerID="cri-o://44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd" gracePeriod=45 Jul 23 19:00:45.396819 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.396795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 19:00:45.487940 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.487904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488144 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.487954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488144 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.487981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9blbm\" (UniqueName: \"kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488144 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488012 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488144 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488144 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm\") pod \"ced595f7-cae1-477b-a1d8-85efedb07e24\" (UID: \"ced595f7-cae1-477b-a1d8-85efedb07e24\") " Jul 23 19:00:45.488477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache" (OuterVolumeSpecName: "model-cache") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:00:45.488477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488275 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home" (OuterVolumeSpecName: "home") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:00:45.488477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:00:45.488477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488428 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-model-cache\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.488477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.488449 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-home\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.490263 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.490218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 19:00:45.490374 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.490279 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm" (OuterVolumeSpecName: "dshm") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 19:00:45.490374 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.490284 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm" (OuterVolumeSpecName: "kube-api-access-9blbm") pod "ced595f7-cae1-477b-a1d8-85efedb07e24" (UID: "ced595f7-cae1-477b-a1d8-85efedb07e24"). InnerVolumeSpecName "kube-api-access-9blbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:00:45.589809 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.589762 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blbm\" (UniqueName: \"kubernetes.io/projected/ced595f7-cae1-477b-a1d8-85efedb07e24-kube-api-access-9blbm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.589809 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.589799 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ced595f7-cae1-477b-a1d8-85efedb07e24-tls-certs\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.589809 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.589813 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-tmp-dir\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.590037 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.589824 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ced595f7-cae1-477b-a1d8-85efedb07e24-dshm\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:00:45.722923 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.722880 2576 generic.go:356] "Generic (PLEG): container finished" podID="ced595f7-cae1-477b-a1d8-85efedb07e24" containerID="44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd" exitCode=0 Jul 23 19:00:45.722923 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.722909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" event={"ID":"ced595f7-cae1-477b-a1d8-85efedb07e24","Type":"ContainerDied","Data":"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd"} Jul 23 19:00:45.722923 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.722920 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" Jul 23 19:00:45.723244 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.722952 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7" event={"ID":"ced595f7-cae1-477b-a1d8-85efedb07e24","Type":"ContainerDied","Data":"db0df2c3330dd03a444e972e204142290f1e71ca3df2f19cd7e266108045c805"} Jul 23 19:00:45.723244 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.722975 2576 scope.go:117] "RemoveContainer" containerID="44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd" Jul 23 19:00:45.732280 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.732263 2576 scope.go:117] "RemoveContainer" containerID="44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd" Jul 23 19:00:45.732518 ip-10-0-135-127 kubenswrapper[2576]: E0723 19:00:45.732500 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd\": container with ID starting with 44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd not found: ID does not exist" containerID="44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd" Jul 23 19:00:45.732565 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.732524 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd"} err="failed to get container status \"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd\": rpc error: code = NotFound desc = could not find container \"44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd\": container with ID starting with 44a22954e8346397d32be1897e86000e9a2ce133a1d45f34c7c3c3db08be98fd not found: ID does not exist" Jul 23 19:00:45.751720 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.751660 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 19:00:45.753736 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.753718 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-60de5a/sub-test-model-60de5a-kserve-57488bb64f-flgk7"] Jul 23 19:00:45.834481 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:45.834452 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced595f7-cae1-477b-a1d8-85efedb07e24" path="/var/lib/kubelet/pods/ced595f7-cae1-477b-a1d8-85efedb07e24/volumes" Jul 23 19:00:50.684735 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:50.684676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:00:50.833320 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:00:50.833291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:01.978055 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:01.978015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:13.963066 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:13.963040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 19:01:13.966775 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:13.966751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 19:01:13.973084 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:13.973060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 19:01:13.976450 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:13.976434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xcgwc_17b9081b-b362-48a0-8fb0-c53eaf57438e/ovn-acl-logging/0.log" Jul 23 19:01:16.013610 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:16.013577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:16.120862 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:16.120835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:24.672487 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:24.672456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:24.773929 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:24.773896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:36.918453 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:36.918417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:37.938310 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:37.938274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:38.771533 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:38.771497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:51.356860 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.356819 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7434-140634787633744"] Jul 23 19:01:51.357365 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.357348 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced595f7-cae1-477b-a1d8-85efedb07e24" containerName="main" Jul 23 19:01:51.357419 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.357366 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced595f7-cae1-477b-a1d8-85efedb07e24" containerName="main" Jul 23 19:01:51.357455 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.357426 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced595f7-cae1-477b-a1d8-85efedb07e24" containerName="main" Jul 23 19:01:51.360062 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.360047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:51.362639 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.362622 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-92csk\"" Jul 23 19:01:51.368715 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.368686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"] Jul 23 19:01:51.522812 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.522780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtd8\" (UniqueName: \"kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8\") pod \"test-curl-7434-140634787633744\" (UID: \"f20f666e-ac04-46e1-83fa-7adce31215b4\") " pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:51.623218 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.623151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtd8\" (UniqueName: \"kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8\") pod \"test-curl-7434-140634787633744\" (UID: \"f20f666e-ac04-46e1-83fa-7adce31215b4\") " pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:51.632495 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.632466 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtd8\" (UniqueName: \"kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8\") pod \"test-curl-7434-140634787633744\" (UID: \"f20f666e-ac04-46e1-83fa-7adce31215b4\") " pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:51.669548 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.669519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:51.796102 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.796080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"] Jul 23 19:01:51.797868 ip-10-0-135-127 kubenswrapper[2576]: W0723 19:01:51.797838 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf20f666e_ac04_46e1_83fa_7adce31215b4.slice/crio-0c16d8deabee354412e719e11d39cd6b8eabad28dbd813f83c922edbd05b58e8 WatchSource:0}: Error finding container 0c16d8deabee354412e719e11d39cd6b8eabad28dbd813f83c922edbd05b58e8: Status 404 returned error can't find the container with id 0c16d8deabee354412e719e11d39cd6b8eabad28dbd813f83c922edbd05b58e8 Jul 23 19:01:51.799309 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:51.799288 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 19:01:52.070615 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:52.070576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787633744" event={"ID":"f20f666e-ac04-46e1-83fa-7adce31215b4","Type":"ContainerStarted","Data":"0c16d8deabee354412e719e11d39cd6b8eabad28dbd813f83c922edbd05b58e8"} Jul 23 19:01:53.077784 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:53.077743 2576 generic.go:356] "Generic (PLEG): container finished" podID="f20f666e-ac04-46e1-83fa-7adce31215b4" containerID="78f762b278678ddd98c50008eb4e2041bb878f62a4ee96a66d2607a95bd4b8c7" exitCode=0 Jul 23 19:01:53.078172 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:53.077837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7434-140634787633744" event={"ID":"f20f666e-ac04-46e1-83fa-7adce31215b4","Type":"ContainerDied","Data":"78f762b278678ddd98c50008eb4e2041bb878f62a4ee96a66d2607a95bd4b8c7"} Jul 23 19:01:54.210608 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.210583 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:54.231700 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.231674 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7434-140634787633744_f20f666e-ac04-46e1-83fa-7adce31215b4/test-curl-7434-140634787633744/0.log" Jul 23 19:01:54.252791 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.252768 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"] Jul 23 19:01:54.255097 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.255076 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7434-140634787633744"] Jul 23 19:01:54.345540 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.345484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vtd8\" (UniqueName: \"kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8\") pod \"f20f666e-ac04-46e1-83fa-7adce31215b4\" (UID: \"f20f666e-ac04-46e1-83fa-7adce31215b4\") " Jul 23 19:01:54.347616 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.347584 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8" (OuterVolumeSpecName: "kube-api-access-9vtd8") pod "f20f666e-ac04-46e1-83fa-7adce31215b4" (UID: "f20f666e-ac04-46e1-83fa-7adce31215b4"). InnerVolumeSpecName "kube-api-access-9vtd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 19:01:54.446998 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:54.446975 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vtd8\" (UniqueName: \"kubernetes.io/projected/f20f666e-ac04-46e1-83fa-7adce31215b4-kube-api-access-9vtd8\") on node \"ip-10-0-135-127.ec2.internal\" DevicePath \"\"" Jul 23 19:01:55.088885 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:55.088857 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:55.088885 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:55.088887 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c16d8deabee354412e719e11d39cd6b8eabad28dbd813f83c922edbd05b58e8" Jul 23 19:01:55.098761 ip-10-0-135-127 kubenswrapper[2576]: E0723 19:01:55.098727 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7434-140634787633744\" is forbidden: User \"system:node:ip-10-0-135-127.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-127.ec2.internal' and this object" podUID="f20f666e-ac04-46e1-83fa-7adce31215b4" pod="opendatahub/test-curl-7434-140634787633744" Jul 23 19:01:55.835910 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:55.835872 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20f666e-ac04-46e1-83fa-7adce31215b4" path="/var/lib/kubelet/pods/f20f666e-ac04-46e1-83fa-7adce31215b4/volumes" Jul 23 19:01:58.128520 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:58.128480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:59.545060 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:59.545025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:01:59.645719 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:01:59.645683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:08.019914 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:08.019886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:12.552132 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:12.552094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:14.651326 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:14.651273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:19.322316 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:19.322259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:19.402029 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:19.401999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:19.632425 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:19.632348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:19.864314 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:19.864279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:20.673076 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:20.673038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:20.766292 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:20.766249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:30.627410 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:30.627373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:35.081918 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:35.081886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:35.489967 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:35.489936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:41.735302 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:41.735263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:42.104723 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:42.104641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:42.205583 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:42.205551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:42.306355 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:42.306327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:42.655373 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:42.655347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54d49797fc-h5jwv_41a8b0d2-6740-40d4-bd35-e5eacaf33d22/authorino/0.log" Jul 23 19:02:48.191000 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:48.190969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z6fnk_243cdeaf-6f1d-4b7d-9512-8962daf175a4/manager/0.log" Jul 23 19:02:48.420368 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:48.420343 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2blvr_ec432d0f-6b9a-45d9-95f2-9f43e82104fe/server/0.log" Jul 23 19:02:48.659971 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:48.659949 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-fjc9v_3d0661c5-019a-48a6-9209-cd00bb3192bb/copy-manifests/0.log" Jul 23 19:02:48.666331 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:48.666312 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-fjc9v_3d0661c5-019a-48a6-9209-cd00bb3192bb/manager/0.log" Jul 23 19:02:50.536468 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.536438 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/util/0.log" Jul 23 19:02:50.542498 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.542476 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/pull/0.log" Jul 23 19:02:50.548450 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.548435 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/extract/0.log" Jul 23 19:02:50.663353 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.663332 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/util/0.log" Jul 23 19:02:50.669568 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.669552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/pull/0.log" Jul 23 19:02:50.675575 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.675561 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/extract/0.log" Jul 23 19:02:50.789684 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.789624 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/extract/0.log" Jul 23 19:02:50.795327 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.795313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/util/0.log" Jul 23 19:02:50.801277 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.801261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/pull/0.log" Jul 23 19:02:50.908678 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.908657 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/util/0.log" Jul 23 19:02:50.914843 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.914821 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/pull/0.log" Jul 23 19:02:50.921288 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:50.921268 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/extract/0.log" Jul 23 19:02:51.034406 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:51.034388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54d49797fc-h5jwv_41a8b0d2-6740-40d4-bd35-e5eacaf33d22/authorino/0.log" Jul 23 19:02:51.148658 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:51.148598 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-m8ddg_271f97f7-ae58-481d-a499-982f72f63c01/manager/0.log" Jul 23 19:02:51.359659 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:51.359636 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jq259_6e56ada6-6109-48e0-9914-98c7698577bc/kuadrant-console-plugin/0.log" Jul 23 19:02:52.174660 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:52.174630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42hzhm_87d25df1-bb18-4b7c-a2eb-3ed44efe72bf/istio-proxy/0.log" Jul 23 19:02:52.894994 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:52.894966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vm7l2_3b1134ff-6dde-4122-bdae-a79bba9cc36e/payload-processing/0.log" Jul 23 19:02:53.017198 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:53.017172 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84b4c5997d-tvj7q_f9200722-028b-4a55-a47f-94b718cbedaa/router/0.log" Jul 23 19:02:53.481993 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:53.481971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-j6rcd_bfddb486-1cc9-4985-8c1d-d0c548de9664/main/0.log" Jul 23 19:02:53.595992 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:53.595969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rrm5v_9b61e365-5215-44cd-b70b-cc7e140e3024/main/0.log" Jul 23 19:02:53.835205 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:53.835146 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-nbdvb_4648a87e-6042-4d90-9b75-7a5dabe3aa53/main/0.log" Jul 23 19:02:53.950449 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:53.950424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5555s_b1e30616-84a2-4b8e-9dcc-7e625d9b6ba2/main/0.log" Jul 23 19:02:59.571558 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:59.571514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:02:59.670799 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:02:59.670768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" limit=1024 Jul 23 19:03:02.721939 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:02.721911 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wdlln_b31242ed-e9d8-4a85-b061-2b226f509c93/global-pull-secret-syncer/0.log" Jul 23 19:03:02.777336 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:02.777306 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7ppq8_b59cdc50-ae0b-4b8e-8741-0dc971d29f91/konnectivity-agent/0.log" Jul 23 19:03:02.871465 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:02.871431 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-127.ec2.internal_8dd7017dd1e272c050bc3c493997c4c9/haproxy/0.log" Jul 23 19:03:07.222564 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.222532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/extract/0.log" Jul 23 19:03:07.245995 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.245961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/util/0.log" Jul 23 19:03:07.272810 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.272788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q687x_ec25ebc5-75f5-4022-a96e-58091705762e/pull/0.log" Jul 23 19:03:07.332097 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.332077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/extract/0.log" Jul 23 19:03:07.352202 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.352180 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/util/0.log" Jul 23 19:03:07.384046 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.384029 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qm7d2_0d63f438-3869-4ae6-bed3-e77a7df90a3e/pull/0.log" Jul 23 19:03:07.415343 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.415313 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/extract/0.log" Jul 23 19:03:07.440283 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.440261 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/util/0.log" Jul 23 19:03:07.461813 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.461796 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z5dqv_e94fe07c-a82e-4232-90ce-b3fd4731b3b2/pull/0.log" Jul 23 19:03:07.494914 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.494861 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/extract/0.log" Jul 23 19:03:07.527547 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.527530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/util/0.log" Jul 23 19:03:07.556050 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.556034 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gsvbd_4a7b28d3-ef3d-448c-ab03-16c7c68ba088/pull/0.log" Jul 23 19:03:07.856842 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.856775 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54d49797fc-h5jwv_41a8b0d2-6740-40d4-bd35-e5eacaf33d22/authorino/0.log" Jul 23 19:03:07.902665 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.902646 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-m8ddg_271f97f7-ae58-481d-a499-982f72f63c01/manager/0.log" Jul 23 19:03:07.977186 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:07.977168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jq259_6e56ada6-6109-48e0-9914-98c7698577bc/kuadrant-console-plugin/0.log" Jul 23 19:03:10.238373 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:10.238342 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9zq6_fa0502df-2c99-4927-bdc7-687173c5a859/node-exporter/0.log" Jul 23 19:03:10.270917 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:10.270894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9zq6_fa0502df-2c99-4927-bdc7-687173c5a859/kube-rbac-proxy/0.log" Jul 23 19:03:10.300262 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:10.300244 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h9zq6_fa0502df-2c99-4927-bdc7-687173c5a859/init-textfile/0.log" Jul 23 19:03:12.263865 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.263833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-n8gtg_45587963-6805-4d17-b098-64a886a354cc/networking-console-plugin/0.log" Jul 23 19:03:12.505390 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.505351 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-djn75"] Jul 23 19:03:12.505761 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.505742 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20f666e-ac04-46e1-83fa-7adce31215b4" containerName="test-curl-7434-140634787633744" Jul 23 19:03:12.505846 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.505763 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20f666e-ac04-46e1-83fa-7adce31215b4" containerName="test-curl-7434-140634787633744" Jul 23 19:03:12.505885 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.505851 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20f666e-ac04-46e1-83fa-7adce31215b4" containerName="test-curl-7434-140634787633744" Jul 23 19:03:12.508857 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.508842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.511477 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.511455 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-48l67\"/\"default-dockercfg-r5r2t\"" Jul 23 19:03:12.511627 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.511477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48l67\"/\"openshift-service-ca.crt\"" Jul 23 19:03:12.511627 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.511518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-48l67\"/\"kube-root-ca.crt\"" Jul 23 19:03:12.515381 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.515327 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-djn75"] Jul 23 19:03:12.648897 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.648867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-lib-modules\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.648897 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.648901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-proc\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.649115 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.648919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-sys\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.649115 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.648935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ddj\" (UniqueName: \"kubernetes.io/projected/821364d9-71cb-422e-af51-56765740e839-kube-api-access-t5ddj\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.649115 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.649010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-podres\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750239 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-lib-modules\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750366 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-proc\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750366 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-sys\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750366 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ddj\" (UniqueName: \"kubernetes.io/projected/821364d9-71cb-422e-af51-56765740e839-kube-api-access-t5ddj\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750366 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-podres\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750569 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-lib-modules\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750569 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-sys\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750569 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-proc\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.750569 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.750441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/821364d9-71cb-422e-af51-56765740e839-podres\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.760957 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.760928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ddj\" (UniqueName: \"kubernetes.io/projected/821364d9-71cb-422e-af51-56765740e839-kube-api-access-t5ddj\") pod \"perf-node-gather-daemonset-djn75\" (UID: \"821364d9-71cb-422e-af51-56765740e839\") " pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.819614 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.819550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:12.896101 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.896070 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/1.log" Jul 23 19:03:12.900895 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.900871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ktvjm_ab6594f7-2fae-4ad8-b3b0-7a08931be111/console-operator/2.log" Jul 23 19:03:12.952074 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:12.952049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-48l67/perf-node-gather-daemonset-djn75"] Jul 23 19:03:12.953593 ip-10-0-135-127 kubenswrapper[2576]: W0723 19:03:12.953571 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod821364d9_71cb_422e_af51_56765740e839.slice/crio-737acb1a1145893162b42cacf1c44615c1baeb2da3cfa9b64958b3376b6cd42e WatchSource:0}: Error finding container 737acb1a1145893162b42cacf1c44615c1baeb2da3cfa9b64958b3376b6cd42e: Status 404 returned error can't find the container with id 737acb1a1145893162b42cacf1c44615c1baeb2da3cfa9b64958b3376b6cd42e Jul 23 19:03:13.438282 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.438252 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76c6d65d8d-x76tf_4c8930c4-753e-48b6-8674-9b86936bbfea/console/0.log" Jul 23 19:03:13.470418 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.470383 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-fk5ck_1b4b3795-c27a-4228-a220-d856254e06dc/download-server/0.log" Jul 23 19:03:13.503464 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.503437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" event={"ID":"821364d9-71cb-422e-af51-56765740e839","Type":"ContainerStarted","Data":"fdf007c483fb995c9a9e9e46c88ee056528af5226715357dd951e4f0b8e67d97"} Jul 23 19:03:13.503586 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.503487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" event={"ID":"821364d9-71cb-422e-af51-56765740e839","Type":"ContainerStarted","Data":"737acb1a1145893162b42cacf1c44615c1baeb2da3cfa9b64958b3376b6cd42e"} Jul 23 19:03:13.503586 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.503519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:13.521269 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:13.521214 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" podStartSLOduration=1.521199235 podStartE2EDuration="1.521199235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 19:03:13.518725757 +0000 UTC m=+2820.249400847" watchObservedRunningTime="2026-07-23 19:03:13.521199235 +0000 UTC m=+2820.251874327" Jul 23 19:03:14.135518 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:14.135490 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-kqvbt_b2904bb2-0993-4eb8-8d07-2194054515f6/volume-data-source-validator/0.log" Jul 23 19:03:15.246784 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:15.246759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rhlsn_4bc9a111-88cb-4371-a4b7-2a8161730ec4/dns/0.log" Jul 23 19:03:15.276688 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:15.276666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rhlsn_4bc9a111-88cb-4371-a4b7-2a8161730ec4/kube-rbac-proxy/0.log" Jul 23 19:03:15.309507 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:15.309487 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bnvn_f169b268-cb81-47b7-a6ae-b4305a26c317/dns-node-resolver/0.log" Jul 23 19:03:15.914757 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:15.914728 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56d5b57745-gcm5f_64cb1b91-c587-4298-94d7-c19c0a495986/registry/0.log" Jul 23 19:03:15.952281 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:15.952258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcl42_6201bed0-3725-46e8-aa9a-25259f731d44/node-ca/0.log" Jul 23 19:03:16.920899 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:16.920859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42hzhm_87d25df1-bb18-4b7c-a2eb-3ed44efe72bf/istio-proxy/0.log" Jul 23 19:03:17.318004 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:17.317968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vm7l2_3b1134ff-6dde-4122-bdae-a79bba9cc36e/payload-processing/0.log" Jul 23 19:03:17.360878 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:17.360848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84b4c5997d-tvj7q_f9200722-028b-4a55-a47f-94b718cbedaa/router/0.log" Jul 23 19:03:18.057977 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:18.057948 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpt6w_104c6971-882c-4298-9de3-7a9f2bf3cc26/serve-healthcheck-canary/0.log" Jul 23 19:03:18.625363 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:18.625288 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4n6f_1bd220f8-70f0-491d-b362-cb4d9db633f4/kube-rbac-proxy/0.log" Jul 23 19:03:18.644798 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:18.644754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4n6f_1bd220f8-70f0-491d-b362-cb4d9db633f4/exporter/0.log" Jul 23 19:03:18.664187 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:18.664162 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s4n6f_1bd220f8-70f0-491d-b362-cb4d9db633f4/extractor/0.log" Jul 23 19:03:19.518984 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:19.518956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-48l67/perf-node-gather-daemonset-djn75" Jul 23 19:03:21.026901 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:21.026870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z6fnk_243cdeaf-6f1d-4b7d-9512-8962daf175a4/manager/0.log" Jul 23 19:03:21.179818 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:21.179794 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2blvr_ec432d0f-6b9a-45d9-95f2-9f43e82104fe/server/0.log" Jul 23 19:03:21.273807 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:21.273777 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-fjc9v_3d0661c5-019a-48a6-9209-cd00bb3192bb/manager/0.log" Jul 23 19:03:21.292444 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:21.292381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77bdbd87db-fjc9v_3d0661c5-019a-48a6-9209-cd00bb3192bb/copy-manifests/0.log" Jul 23 19:03:22.807278 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:22.807184 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-778f8cb4df-gcq9r_8d315cd6-9cc2-4df2-864d-bcd1f7667b2d/manager/0.log" Jul 23 19:03:23.272968 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:23.272941 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42hzhm_87d25df1-bb18-4b7c-a2eb-3ed44efe72bf/istio-proxy/0.log" Jul 23 19:03:23.499805 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:23.499776 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vm7l2_3b1134ff-6dde-4122-bdae-a79bba9cc36e/payload-processing/0.log" Jul 23 19:03:23.519169 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:23.519144 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84b4c5997d-tvj7q_f9200722-028b-4a55-a47f-94b718cbedaa/router/0.log" Jul 23 19:03:24.701773 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:24.701741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf42hzhm_87d25df1-bb18-4b7c-a2eb-3ed44efe72bf/istio-proxy/0.log" Jul 23 19:03:24.948987 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:24.948961 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vm7l2_3b1134ff-6dde-4122-bdae-a79bba9cc36e/payload-processing/0.log" Jul 23 19:03:24.963084 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:24.963019 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84b4c5997d-tvj7q_f9200722-028b-4a55-a47f-94b718cbedaa/router/0.log" Jul 23 19:03:27.759428 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:27.759401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-phlf4_03025fe2-d410-4448-b76a-b271b2335c8f/migrator/0.log" Jul 23 19:03:27.779236 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:27.779200 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-phlf4_03025fe2-d410-4448-b76a-b271b2335c8f/graceful-termination/0.log" Jul 23 19:03:29.467643 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.467613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/kube-multus-additional-cni-plugins/0.log" Jul 23 19:03:29.487682 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.487659 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/egress-router-binary-copy/0.log" Jul 23 19:03:29.509385 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.509354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/cni-plugins/0.log" Jul 23 19:03:29.529774 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.529754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/bond-cni-plugin/0.log" Jul 23 19:03:29.549285 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.549265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/routeoverride-cni/0.log" Jul 23 19:03:29.569878 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.569859 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/whereabouts-cni-bincopy/0.log" Jul 23 19:03:29.591328 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.591304 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vrf52_a4ec536d-8503-4045-bae0-908bd6c9f99a/whereabouts-cni/0.log" Jul 23 19:03:29.689161 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.689136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zw522_4952ba4f-813b-43db-b65f-abba33ad4d2a/kube-multus/0.log" Jul 23 19:03:29.804391 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.804321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wc7ll_17732286-b41e-4c5d-913a-440a7677d70d/network-metrics-daemon/0.log" Jul 23 19:03:29.823652 ip-10-0-135-127 kubenswrapper[2576]: I0723 19:03:29.823630 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wc7ll_17732286-b41e-4c5d-913a-440a7677d70d/kube-rbac-proxy/0.log"