Jul 06 20:12:37.852453 ip-10-0-130-104 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 20:12:37.852467 ip-10-0-130-104 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 20:12:37.852476 ip-10-0-130-104 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 20:12:37.852893 ip-10-0-130-104 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 20:12:48.081384 ip-10-0-130-104 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 20:12:48.081402 ip-10-0-130-104 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 960c4fa5d4e941a3a964b98c45ef5d3c -- Jul 06 20:14:42.152873 ip-10-0-130-104 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.566226 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572651 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572662 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572665 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572670 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572673 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572676 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572678 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:42.701320 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572681 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:42.646653 ip-10-0-130-104 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572684 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572686 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572689 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572697 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572700 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572703 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572707 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572711 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572714 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572716 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572719 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572722 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572724 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572728 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572732 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572735 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572737 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572740 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:42.702472 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572743 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572746 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572748 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572752 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572756 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572759 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572762 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572764 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572767 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572769 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572772 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572774 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572777 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572779 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572782 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572784 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572787 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572790 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572792 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572796 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:42.703081 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572799 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572803 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572806 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572809 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572811 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572814 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572817 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572819 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572821 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572824 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572826 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572828 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572831 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572834 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572836 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572840 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572842 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572845 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572847 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572850 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:42.703712 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572852 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572855 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572857 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572860 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572862 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572865 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572868 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572870 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572873 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572875 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572877 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572880 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572883 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572885 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572888 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572890 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572893 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572895 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572898 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572900 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:42.704335 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572902 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572905 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572907 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572910 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572912 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572915 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572917 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572920 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572923 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572926 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572928 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572930 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.572933 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573612 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573632 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573635 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573638 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573640 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573643 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573646 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:42.705045 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573649 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573651 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573654 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573656 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573660 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573663 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573666 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573668 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573671 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573674 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573676 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573679 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573681 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573683 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573686 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573689 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573691 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573694 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573696 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573699 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:42.705719 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573702 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573704 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573707 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573709 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573712 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573715 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573718 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573720 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573723 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573725 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573728 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573730 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573732 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573735 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573738 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573740 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573742 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573745 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573748 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:42.706425 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573751 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573756 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573760 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573763 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573766 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573768 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573771 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573775 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573778 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573780 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573783 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573786 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573788 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573792 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573795 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573798 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573801 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573804 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573806 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:42.707113 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573809 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573812 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573814 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573817 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573819 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573822 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573824 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573827 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573830 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573832 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573835 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573837 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573840 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573844 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573847 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573850 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573852 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573857 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573860 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:42.707779 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573865 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573868 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573871 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573874 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573877 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573879 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573882 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573885 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573887 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573890 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573893 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573896 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573899 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573902 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.573905 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574252 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574256 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574259 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574261 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574264 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:42.708325 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574267 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574269 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574272 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574278 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574281 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574285 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574288 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574290 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574293 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574296 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574299 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574301 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574305 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574308 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574310 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574313 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574315 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574318 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574320 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574323 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:42.708974 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574326 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574328 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574331 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574333 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574336 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574338 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574341 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574343 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574346 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574348 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574351 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574353 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574355 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574358 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574360 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574364 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574366 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574369 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574374 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:42.709580 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574377 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574379 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574382 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574385 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574388 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574392 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574394 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574397 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574399 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574402 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574404 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574406 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574409 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574412 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574415 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574418 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574420 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574423 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574425 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574428 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:42.710519 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574430 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574432 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574435 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574438 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574442 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574445 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574448 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574450 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574455 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574458 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574462 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574465 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574468 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574470 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574473 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574475 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574477 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574481 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574483 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:42.711388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574486 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574488 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574491 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574493 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574496 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574498 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574500 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574503 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574506 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574509 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574511 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574514 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574516 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574518 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574521 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.574523 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574587 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574594 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574600 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574604 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574608 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:14:42.712269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574612 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574616 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574634 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574638 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574641 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574648 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574652 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574655 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574658 2574 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574662 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574665 2574 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574668 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574670 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574675 2574 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574678 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574681 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574684 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574687 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574691 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574694 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574696 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574700 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574703 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574706 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:14:42.713134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574709 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574712 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574714 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574718 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574721 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574724 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574727 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574730 2574 flags.go:64] FLAG: --enable-server="true" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574734 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574738 2574 flags.go:64] FLAG: --event-burst="100" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574741 2574 flags.go:64] FLAG: --event-qps="50" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574744 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574747 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574750 2574 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574754 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574757 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574760 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574764 2574 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574767 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574770 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574772 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574776 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574778 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574781 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574784 2574 flags.go:64] FLAG: --feature-gates="" Jul 06 20:14:42.714244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574788 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574791 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574794 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574797 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574799 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574802 2574 flags.go:64] FLAG: --help="false" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574805 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-104.ec2.internal" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574808 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574811 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574814 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574817 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574820 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574823 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574826 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574828 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574832 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574835 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574838 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574841 2574 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574844 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574846 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574849 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574852 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574856 2574 flags.go:64] FLAG: --lock-file="" Jul 06 20:14:42.715334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574859 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574862 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574865 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574870 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574873 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574875 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574879 2574 flags.go:64] FLAG: --logging-format="text" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574882 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574885 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574888 2574 flags.go:64] FLAG: --manifest-url="" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574890 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574894 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574897 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574901 2574 flags.go:64] FLAG: --max-pods="110" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574904 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574907 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574910 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574912 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574915 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574918 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574921 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574927 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574930 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574934 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:14:42.716432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574937 2574 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574940 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574945 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574948 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574951 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574954 2574 flags.go:64] FLAG: --port="10250" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574957 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574960 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01212364aa261a6db" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574965 2574 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574968 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574970 2574 flags.go:64] FLAG: --register-node="true" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574973 2574 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574977 2574 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574980 2574 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574982 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574985 2574 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574989 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574992 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574994 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.574997 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575000 2574 flags.go:64] FLAG: --runonce="false" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575003 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575006 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575008 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575011 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:14:42.887290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575014 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575017 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575020 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575023 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575026 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575028 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575031 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575036 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575039 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575042 2574 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575044 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575050 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575053 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575055 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575061 2574 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575065 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575068 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575071 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575074 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575077 2574 flags.go:64] FLAG: --v="2" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575080 2574 flags.go:64] FLAG: --version="false" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575084 2574 flags.go:64] FLAG: --vmodule="" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575087 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.575090 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575199 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:42.907487 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575203 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575206 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575209 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575212 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575215 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575218 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575221 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575224 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575227 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575229 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575232 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575234 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575237 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575239 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575243 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575245 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575248 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575250 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575253 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575255 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:42.908990 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575258 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575260 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575264 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575267 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575270 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575274 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575278 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575281 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575284 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575287 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575290 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575293 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575295 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575298 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575300 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575303 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575306 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575308 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575311 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:42.909831 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575313 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575316 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575318 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575321 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575323 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575327 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575330 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575334 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575338 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575340 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575343 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575345 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575349 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575353 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575355 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575359 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575362 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575366 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575369 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:42.912310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575371 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575374 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575377 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575380 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575382 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575385 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575387 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575390 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575392 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575395 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575397 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575400 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575403 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575405 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575407 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575410 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575413 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575415 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575418 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575420 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:42.913085 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575424 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575426 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575429 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575432 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575434 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575437 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575439 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575442 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575446 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575449 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575451 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575457 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575459 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575462 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575465 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575467 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575470 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575472 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575475 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:42.913652 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.575478 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.576118 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.583798 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.583815 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.584712 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.584720 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584778 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584783 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584786 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584789 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584792 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584794 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584799 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584803 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:42.914212 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584805 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584808 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584811 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584813 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584816 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584819 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584822 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584824 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584827 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584830 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584833 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584836 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584839 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584843 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584846 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584849 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584853 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584856 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584858 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584861 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:42.914629 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584864 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584866 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584870 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584872 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584875 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584878 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584881 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584883 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584886 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584889 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584891 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584894 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584896 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584899 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584902 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584904 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584907 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584909 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584912 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584914 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:42.915209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584917 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584920 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584922 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584925 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584927 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584930 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584932 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584935 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584938 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584941 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584944 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584946 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584949 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584951 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584954 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584956 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584961 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584963 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584967 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584970 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:42.915773 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584973 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584990 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584994 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.584997 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585000 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585002 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585005 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585008 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585011 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585014 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585017 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585021 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585024 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585027 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585030 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585033 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585035 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585038 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585041 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585043 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:44.355699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585045 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585048 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585050 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585053 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585056 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585058 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585060 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585063 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585070 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585072 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585075 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.585080 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585188 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585193 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585196 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:14:44.357019 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585199 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585202 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585205 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585207 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585210 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585216 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585218 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585221 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585224 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585227 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585230 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585233 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585236 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585238 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585241 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585243 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585246 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585248 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585251 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585254 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:14:44.357585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585256 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585258 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585261 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585264 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585266 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585270 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585272 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585275 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585278 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585280 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585283 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585285 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585288 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585290 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585293 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585295 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585298 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585301 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585304 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585307 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:14:44.358176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585309 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585312 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585316 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585318 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585322 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585326 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585329 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585331 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585334 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585338 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585341 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585344 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585346 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585349 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585351 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585354 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585356 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585360 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585362 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:14:44.358754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585366 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585369 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585372 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585374 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585377 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585379 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585382 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585384 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585387 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585389 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585393 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585396 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585398 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585402 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585405 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585408 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585412 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585415 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585418 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:14:44.359296 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585421 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585424 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585426 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585429 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585431 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585434 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585436 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585439 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585441 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585444 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585446 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585450 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585452 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585455 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585457 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585460 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585462 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:14:44.359834 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:42.585465 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.585470 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.585585 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.589136 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.590891 2574 server.go:1019] "Starting client certificate rotation" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.590986 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.591021 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.593638 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.596487 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.611094 2574 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.613212 2574 log.go:25] "Validated CRI v1 image API" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.614017 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:14:44.362409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.624599 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dc11fd49-2914-4b8b-bdb9-6b37747cfb19:/dev/nvme0n1p4 fd8908d6-2a5e-44fe-b905-29bcc9fe71ad:/dev/nvme0n1p3] Jul 06 20:14:44.363213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.624631 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:14:44.363213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.627289 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.628854 2574 manager.go:216] Machine: {Timestamp:2026-07-06 20:14:42.62838165 +0000 UTC m=+0.353287260 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100019 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec205c4796f9c6e5163380fa0b8ae104 SystemUUID:ec205c47-96f9-c6e5-1633-80fa0b8ae104 BootID:960c4fa5-d4e9-41a3-a964-b98c45ef5d3c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:34:91:f7:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:34:91:f7:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:46:68:1a:a6:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.628945 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.629042 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.630173 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.630195 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.630894 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.630905 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.630912 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.631536 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.632730 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635438 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635545 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635568 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635577 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635589 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635598 2574 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635607 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:14:44.363575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635919 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.635935 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.638635 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.640389 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.640416 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641823 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641836 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641842 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641847 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641855 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641861 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641868 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641873 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641879 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641884 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641892 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.641897 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.643062 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.643069 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.645918 2574 server.go:1268] "Started kubelet" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.646002 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.646122 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.646725 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.647643 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.648694 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.651081 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653184 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653188 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.653303 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653752 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653766 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653878 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.653991 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.654000 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.654118 2574 factory.go:55] Registering systemd factory Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.654130 2574 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.655144 2574 factory.go:153] Registering CRI-O factory Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.655160 2574 factory.go:223] Registration of the crio container factory successfully Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.655229 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.655266 2574 factory.go:103] Registering Raw factory Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.655286 2574 manager.go:1196] Started watching for new ooms in manager Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.656456 2574 manager.go:349] Starting recovery of all containers Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.658257 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4v89n" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.658416 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.658475 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.658601 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:14:44.364453 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.659011 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.659177 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.658994 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-104.ec2.internal.18bfcc8cd197ffd4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-104.ec2.internal,UID:ip-10-0-130-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-104.ec2.internal,},FirstTimestamp:2026-07-06 20:14:42.645893076 +0000 UTC m=+0.370798687,LastTimestamp:2026-07-06 20:14:42.645893076 +0000 UTC m=+0.370798687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-104.ec2.internal,}" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.666475 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4v89n" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.671591 2574 manager.go:354] Recovery completed Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.675600 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677364 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677390 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677400 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677952 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677960 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.677974 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.678007 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.678732 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.678753 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.678769 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.679238 2574 policy_none.go:49] "None policy: Start" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.679253 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.679265 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.680371 2574 policy_none.go:47] "Start" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.683315 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-104.ec2.internal.18bfcc8cd378689b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-104.ec2.internal,UID:ip-10-0-130-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-104.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-104.ec2.internal,},FirstTimestamp:2026-07-06 20:14:42.677377179 +0000 UTC m=+0.402282793,LastTimestamp:2026-07-06 20:14:42.677377179 +0000 UTC m=+0.402282793,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-104.ec2.internal,}" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722200 2574 manager.go:337] "Starting Device Plugin manager" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.722355 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722368 2574 server.go:87] "Starting device plugin registration server" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722555 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722564 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722709 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722778 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.722784 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.723192 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.723588 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.822805 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.824189 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.365740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.824211 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.824221 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.824250 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.825578 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.831058 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.831090 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.831110 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.831141 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.833290 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.838163 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.838181 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-104.ec2.internal\": node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.877912 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.931785 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal"] Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.931843 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.933203 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.933224 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.933234 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.935354 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.935473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.935496 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936020 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936030 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936050 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936062 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936078 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.936089 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.938129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.938149 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.938808 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.938829 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.938839 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.956155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5457eba2732788e8cf4686bd84b1d393-config\") pod \"kube-apiserver-proxy-ip-10-0-130-104.ec2.internal\" (UID: \"5457eba2732788e8cf4686bd84b1d393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.956192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:42.956223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.974243 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-104.ec2.internal\" not found" node="ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.555532 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.978057 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:42.978506 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-104.ec2.internal\" not found" node="ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5457eba2732788e8cf4686bd84b1d393-config\") pod \"kube-apiserver-proxy-ip-10-0-130-104.ec2.internal\" (UID: \"5457eba2732788e8cf4686bd84b1d393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5457eba2732788e8cf4686bd84b1d393-config\") pod \"kube-apiserver-proxy-ip-10-0-130-104.ec2.internal\" (UID: \"5457eba2732788e8cf4686bd84b1d393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.291240 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37165545d2531421ef847dfe3fcc003e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal\" (UID: \"37165545d2531421ef847dfe3fcc003e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.313111 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.413349 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.510789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.513879 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.557082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.515420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:44.779459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.615123 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.779459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.715574 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.800095 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.817151 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.824748 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.825415 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.825441 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.901090 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.906659 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:09:43 +0000 UTC" deadline="2028-02-01 07:26:16.456092494 +0000 UTC" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:44.906687 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13787h11m31.549408079s" Jul 06 20:14:44.967791 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:44.917941 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.018608 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.096126 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.097022 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.097042 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.097051 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:14:45.132480 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.118786 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:45.259917 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:45.208572 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37165545d2531421ef847dfe3fcc003e.slice/crio-a56007cf3d65c107f414057e7d112faa3a214c17eddc6e2bb893bb221c84ddc9 WatchSource:0}: Error finding container a56007cf3d65c107f414057e7d112faa3a214c17eddc6e2bb893bb221c84ddc9: Status 404 returned error can't find the container with id a56007cf3d65c107f414057e7d112faa3a214c17eddc6e2bb893bb221c84ddc9 Jul 06 20:14:45.259917 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.219555 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-104.ec2.internal\" not found" Jul 06 20:14:45.259917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.219619 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:14:45.259917 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:45.222939 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5457eba2732788e8cf4686bd84b1d393.slice/crio-45a19290b697c5d37bba7720c3cf7d9879b30b10adbff097b236635e6e59a232 WatchSource:0}: Error finding container 45a19290b697c5d37bba7720c3cf7d9879b30b10adbff097b236635e6e59a232: Status 404 returned error can't find the container with id 45a19290b697c5d37bba7720c3cf7d9879b30b10adbff097b236635e6e59a232 Jul 06 20:14:45.477003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.307120 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:45.477003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.388638 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" Jul 06 20:14:45.477003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.400470 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:14:45.477003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.401615 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" Jul 06 20:14:45.477003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.410969 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:14:45.689666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.631828 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.872575 2574 apiserver.go:52] "Watching apiserver" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.879306 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.879676 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mpq6z","kube-system/konnectivity-agent-5clp9","kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg","openshift-image-registry/node-ca-4djqt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal","openshift-multus/multus-additional-cni-plugins-rm2rx","openshift-multus/multus-cmtzl","openshift-cluster-node-tuning-operator/tuned-nrf76","openshift-multus/network-metrics-daemon-d9crx","openshift-network-diagnostics/network-check-target-5hbp9","openshift-network-operator/iptables-alerter-qlmgh","openshift-ovn-kubernetes/ovnkube-node-6vmlz"] Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.882277 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.885820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.887333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gqpc7\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.887557 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.887914 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.887921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.888058 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.888140 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.888370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.888433 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.890691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.893311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.893393 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.895424 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.896776 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.896881 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.897990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898030 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898104 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9p58p\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.897989 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898282 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898354 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmtzl" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.898355 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b5xbg\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.900861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.902243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zlqph\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.902452 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.902667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-58zgd\"" Jul 06 20:14:45.903321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.903108 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.903348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.903412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.903994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zt57m\"" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.904038 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.905786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:45.906652 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:45.905834 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:45.909540 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.908862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:45.909540 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-os-release\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-sys-fs\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e59897e4-965e-4df8-899d-70fcfe6d1ed8-serviceca\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-socket-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-registration-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wrd\" (UniqueName: \"kubernetes.io/projected/e59897e4-965e-4df8-899d-70fcfe6d1ed8-kube-api-access-85wrd\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-binary-copy\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fb82de6-3f26-4671-89d1-4b13446422bf-agent-certs\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-device-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-etc-selinux\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-system-cni-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx44f\" (UniqueName: \"kubernetes.io/projected/9b7ec425-fa4f-4648-8e08-561cb0cb7984-kube-api-access-vx44f\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fb82de6-3f26-4671-89d1-4b13446422bf-konnectivity-ca\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:45.912351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.909995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.910018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sjlm\" (UniqueName: \"kubernetes.io/projected/3a017b21-d982-4533-bea5-899975060759-kube-api-access-4sjlm\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.910038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e59897e4-965e-4df8-899d-70fcfe6d1ed8-host\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.910064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cnibin\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.911470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.911668 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.913684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.913347 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8qhr\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.914037 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916133 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916441 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916518 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916443 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-82vcn\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.916838 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.917045 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:14:45.917220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.917058 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:14:45.921069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.919778 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:14:45.986553 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.956485 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xlmfs" Jul 06 20:14:45.986553 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.964584 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xlmfs" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:45.989632 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-var-lib-kubelet\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-systemd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-dbus\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-socket-dir-parent\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-netns\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-etc-kubernetes\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e59897e4-965e-4df8-899d-70fcfe6d1ed8-host\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-conf\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cnibin\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-log-socket\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-netd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-host\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wrd\" (UniqueName: \"kubernetes.io/projected/e59897e4-965e-4df8-899d-70fcfe6d1ed8-kube-api-access-85wrd\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.060594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-lib-modules\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-node-log\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-multus-certs\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.010969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-sys\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-etc-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-k8s-cni-cncf-io\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011175 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cnibin\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e59897e4-965e-4df8-899d-70fcfe6d1ed8-serviceca\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e59897e4-965e-4df8-899d-70fcfe6d1ed8-host\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-systemd-units\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovn-node-metrics-cert\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnz59\" (UniqueName: \"kubernetes.io/projected/efc56e0e-e63c-4976-b894-0775506ea70b-kube-api-access-jnz59\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxsf\" (UniqueName: \"kubernetes.io/projected/de5a9039-dd79-406d-96c8-0e10737ecf6d-kube-api-access-7sxsf\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-run\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.061445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-multus\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-kubelet\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-device-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-etc-selinux\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-var-lib-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-ovn\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-env-overrides\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8rk\" (UniqueName: \"kubernetes.io/projected/1a4e3e5f-2493-490f-bbee-79748e604a3f-kube-api-access-mq8rk\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-kubelet-config\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-systemd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-system-cni-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-kubelet\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-slash\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011916 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e59897e4-965e-4df8-899d-70fcfe6d1ed8-serviceca\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.063750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.011957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-device-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-conf-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-os-release\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-sys-fs\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-cnibin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-multus-daemon-config\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-modprobe-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysconfig\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38fdc5ad-4828-437f-81fa-8640d7a07e02-host-slash\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-bin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-kubernetes\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-tmp\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.064791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38fdc5ad-4828-437f-81fa-8640d7a07e02-iptables-alerter-script\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhx5\" (UniqueName: \"kubernetes.io/projected/38fdc5ad-4828-437f-81fa-8640d7a07e02-kube-api-access-qkhx5\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-binary-copy\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-etc-selinux\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-system-cni-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fb82de6-3f26-4671-89d1-4b13446422bf-agent-certs\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012716 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-os-release\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-netns\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012850 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-sys-fs\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-config\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.012948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-system-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013091 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b7ec425-fa4f-4648-8e08-561cb0cb7984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-hostroot\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013229 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:14:46.065839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx44f\" (UniqueName: \"kubernetes.io/projected/9b7ec425-fa4f-4648-8e08-561cb0cb7984-kube-api-access-vx44f\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fb82de6-3f26-4671-89d1-4b13446422bf-konnectivity-ca\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sjlm\" (UniqueName: \"kubernetes.io/projected/3a017b21-d982-4533-bea5-899975060759-kube-api-access-4sjlm\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-os-release\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-cni-binary-copy\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-tuned\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-bin\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ddd\" (UniqueName: \"kubernetes.io/projected/edf2b52e-ccb9-483b-a378-5de693c9fbc2-kube-api-access-67ddd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-socket-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-registration-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-script-lib\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.013894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-socket-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.014000 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a017b21-d982-4533-bea5-899975060759-registration-dir\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.014254 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b7ec425-fa4f-4648-8e08-561cb0cb7984-cni-binary-copy\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.014545 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fb82de6-3f26-4671-89d1-4b13446422bf-konnectivity-ca\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.017410 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fb82de6-3f26-4671-89d1-4b13446422bf-agent-certs\") pod \"konnectivity-agent-5clp9\" (UID: \"7fb82de6-3f26-4671-89d1-4b13446422bf\") " pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:46.066928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.021502 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wrd\" (UniqueName: \"kubernetes.io/projected/e59897e4-965e-4df8-899d-70fcfe6d1ed8-kube-api-access-85wrd\") pod \"node-ca-4djqt\" (UID: \"e59897e4-965e-4df8-899d-70fcfe6d1ed8\") " pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.067773 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.023181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sjlm\" (UniqueName: \"kubernetes.io/projected/3a017b21-d982-4533-bea5-899975060759-kube-api-access-4sjlm\") pod \"aws-ebs-csi-driver-node-76tlg\" (UID: \"3a017b21-d982-4533-bea5-899975060759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.067773 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.024770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx44f\" (UniqueName: \"kubernetes.io/projected/9b7ec425-fa4f-4648-8e08-561cb0cb7984-kube-api-access-vx44f\") pod \"multus-additional-cni-plugins-rm2rx\" (UID: \"9b7ec425-fa4f-4648-8e08-561cb0cb7984\") " pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.070794 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.070757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" event={"ID":"5457eba2732788e8cf4686bd84b1d393","Type":"ContainerStarted","Data":"45a19290b697c5d37bba7720c3cf7d9879b30b10adbff097b236635e6e59a232"} Jul 06 20:14:46.072329 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.072304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" event={"ID":"37165545d2531421ef847dfe3fcc003e","Type":"ContainerStarted","Data":"a56007cf3d65c107f414057e7d112faa3a214c17eddc6e2bb893bb221c84ddc9"} Jul 06 20:14:46.114995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.114971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-script-lib\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-var-lib-kubelet\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-systemd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-dbus\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-socket-dir-parent\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115125 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-netns\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-etc-kubernetes\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-conf\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-log-socket\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-netd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-host\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-lib-modules\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-node-log\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-multus-certs\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-sys\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-etc-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-k8s-cni-cncf-io\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-systemd-units\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovn-node-metrics-cert\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnz59\" (UniqueName: \"kubernetes.io/projected/efc56e0e-e63c-4976-b894-0775506ea70b-kube-api-access-jnz59\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxsf\" (UniqueName: \"kubernetes.io/projected/de5a9039-dd79-406d-96c8-0e10737ecf6d-kube-api-access-7sxsf\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-run\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115587 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-multus-certs\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-multus\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.115682 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:46.115705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-multus\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-kubelet\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115760 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-kubelet\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.115764 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:46.615731489 +0000 UTC m=+3.106355031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-var-lib-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115818 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-sys\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-ovn\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-env-overrides\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8rk\" (UniqueName: \"kubernetes.io/projected/1a4e3e5f-2493-490f-bbee-79748e604a3f-kube-api-access-mq8rk\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-var-lib-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-kubelet-config\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-systemd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115966 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-dbus\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-kubelet\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-var-lib-kubelet\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-slash\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116075 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-slash\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.116520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-systemd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116127 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-etc-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116130 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-socket-dir-parent\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115400 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-etc-kubernetes\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116163 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-conf\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116178 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-netd\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-log-socket\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-conf-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-cnibin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.116299 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-multus-daemon-config\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.116343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:14:46.616327627 +0000 UTC m=+3.106951162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-modprobe-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysconfig\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.117322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38fdc5ad-4828-437f-81fa-8640d7a07e02-host-slash\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-bin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-kubernetes\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-tmp\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38fdc5ad-4828-437f-81fa-8640d7a07e02-iptables-alerter-script\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhx5\" (UniqueName: \"kubernetes.io/projected/38fdc5ad-4828-437f-81fa-8640d7a07e02-kube-api-access-qkhx5\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-netns\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-config\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-system-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116671 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-script-lib\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-hostroot\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-os-release\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116732 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-host\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-cni-binary-copy\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-tuned\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116799 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-multus-daemon-config\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-systemd-units\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-bin\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.118115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ddd\" (UniqueName: \"kubernetes.io/projected/edf2b52e-ccb9-483b-a378-5de693c9fbc2-kube-api-access-67ddd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116850 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-openvswitch\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116931 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-lib-modules\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116952 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-run\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116978 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-node-log\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.116990 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117041 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysconfig\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117073 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-modprobe-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117125 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-sysctl-d\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.115867 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-run-ovn\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117205 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-netns\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-kubelet\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117245 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-run-netns\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117253 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-cnibin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117288 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-multus-conf-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117308 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-os-release\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117361 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-env-overrides\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117399 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a4e3e5f-2493-490f-bbee-79748e604a3f-host-cni-bin\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117436 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-kubernetes\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-systemd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-hostroot\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117591 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5fd1d381-faee-4d65-9d12-05c0a91abc96-kubelet-config\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117634 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-system-cni-dir\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117665 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-run-k8s-cni-cncf-io\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117806 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efc56e0e-e63c-4976-b894-0775506ea70b-cni-binary-copy\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117854 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efc56e0e-e63c-4976-b894-0775506ea70b-host-var-lib-cni-bin\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.118144 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38fdc5ad-4828-437f-81fa-8640d7a07e02-iptables-alerter-script\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.117253 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38fdc5ad-4828-437f-81fa-8640d7a07e02-host-slash\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.119807 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.118661 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovnkube-config\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.120564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.120518 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-etc-tuned\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.120645 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.120586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a4e3e5f-2493-490f-bbee-79748e604a3f-ovn-node-metrics-cert\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.120698 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.120682 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edf2b52e-ccb9-483b-a378-5de693c9fbc2-tmp\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.128067 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.128042 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:14:46.128067 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.128066 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:14:46.128244 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.128079 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:46.128244 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.128132 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:46.628117522 +0000 UTC m=+3.118741062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:46.130957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.130892 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8rk\" (UniqueName: \"kubernetes.io/projected/1a4e3e5f-2493-490f-bbee-79748e604a3f-kube-api-access-mq8rk\") pod \"ovnkube-node-6vmlz\" (UID: \"1a4e3e5f-2493-490f-bbee-79748e604a3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.131420 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.131386 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxsf\" (UniqueName: \"kubernetes.io/projected/de5a9039-dd79-406d-96c8-0e10737ecf6d-kube-api-access-7sxsf\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.131681 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.131662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhx5\" (UniqueName: \"kubernetes.io/projected/38fdc5ad-4828-437f-81fa-8640d7a07e02-kube-api-access-qkhx5\") pod \"iptables-alerter-qlmgh\" (UID: \"38fdc5ad-4828-437f-81fa-8640d7a07e02\") " pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.131965 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.131945 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ddd\" (UniqueName: \"kubernetes.io/projected/edf2b52e-ccb9-483b-a378-5de693c9fbc2-kube-api-access-67ddd\") pod \"tuned-nrf76\" (UID: \"edf2b52e-ccb9-483b-a378-5de693c9fbc2\") " pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.133288 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.133149 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnz59\" (UniqueName: \"kubernetes.io/projected/efc56e0e-e63c-4976-b894-0775506ea70b-kube-api-access-jnz59\") pod \"multus-cmtzl\" (UID: \"efc56e0e-e63c-4976-b894-0775506ea70b\") " pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.201880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.201856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" Jul 06 20:14:46.207786 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.207698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:46.212020 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.211999 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7ec425_fa4f_4648_8e08_561cb0cb7984.slice/crio-4aae19429576eda34fd8e9db263640616635a6fa3b1edc59261df3ab4aa7da61 WatchSource:0}: Error finding container 4aae19429576eda34fd8e9db263640616635a6fa3b1edc59261df3ab4aa7da61: Status 404 returned error can't find the container with id 4aae19429576eda34fd8e9db263640616635a6fa3b1edc59261df3ab4aa7da61 Jul 06 20:14:46.217245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.217222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" Jul 06 20:14:46.217513 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.217491 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb82de6_3f26_4671_89d1_4b13446422bf.slice/crio-e92306c47a07eefc4ff75c9c49d9a2a74eba9ce15f2953a2bf4e638db2e173b1 WatchSource:0}: Error finding container e92306c47a07eefc4ff75c9c49d9a2a74eba9ce15f2953a2bf4e638db2e173b1: Status 404 returned error can't find the container with id e92306c47a07eefc4ff75c9c49d9a2a74eba9ce15f2953a2bf4e638db2e173b1 Jul 06 20:14:46.226989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.226971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4djqt" Jul 06 20:14:46.227639 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.227611 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a017b21_d982_4533_bea5_899975060759.slice/crio-15653faebf0eae1dfcaa2de3c1f32a4f23338e4cea90e5ad2c5898c2e33c757a WatchSource:0}: Error finding container 15653faebf0eae1dfcaa2de3c1f32a4f23338e4cea90e5ad2c5898c2e33c757a: Status 404 returned error can't find the container with id 15653faebf0eae1dfcaa2de3c1f32a4f23338e4cea90e5ad2c5898c2e33c757a Jul 06 20:14:46.234809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.234751 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmtzl" Jul 06 20:14:46.236192 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.236172 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59897e4_965e_4df8_899d_70fcfe6d1ed8.slice/crio-a9401973c0d522fd3034f7f8b344fa4b940e3d2a34f5a35e589de55b4eeb4a15 WatchSource:0}: Error finding container a9401973c0d522fd3034f7f8b344fa4b940e3d2a34f5a35e589de55b4eeb4a15: Status 404 returned error can't find the container with id a9401973c0d522fd3034f7f8b344fa4b940e3d2a34f5a35e589de55b4eeb4a15 Jul 06 20:14:46.241531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.241511 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrf76" Jul 06 20:14:46.243713 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.243690 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc56e0e_e63c_4976_b894_0775506ea70b.slice/crio-ef573119a9e5ca914e6f0df0134919306fc9c090a5ffdcf0ba47d7f441bce94e WatchSource:0}: Error finding container ef573119a9e5ca914e6f0df0134919306fc9c090a5ffdcf0ba47d7f441bce94e: Status 404 returned error can't find the container with id ef573119a9e5ca914e6f0df0134919306fc9c090a5ffdcf0ba47d7f441bce94e Jul 06 20:14:46.251070 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.251042 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf2b52e_ccb9_483b_a378_5de693c9fbc2.slice/crio-9e029d17cd5a0167c14755ffdf3310b1c1392055203eb056b689559c89ce7561 WatchSource:0}: Error finding container 9e029d17cd5a0167c14755ffdf3310b1c1392055203eb056b689559c89ce7561: Status 404 returned error can't find the container with id 9e029d17cd5a0167c14755ffdf3310b1c1392055203eb056b689559c89ce7561 Jul 06 20:14:46.252397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.252378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qlmgh" Jul 06 20:14:46.260483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.260455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:14:46.262039 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.262015 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fdc5ad_4828_437f_81fa_8640d7a07e02.slice/crio-f13ff712808ce99a0721bdccf6e296e71c6b1ea823b80ebd0b652e6f8ebe5f3d WatchSource:0}: Error finding container f13ff712808ce99a0721bdccf6e296e71c6b1ea823b80ebd0b652e6f8ebe5f3d: Status 404 returned error can't find the container with id f13ff712808ce99a0721bdccf6e296e71c6b1ea823b80ebd0b652e6f8ebe5f3d Jul 06 20:14:46.270378 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:14:46.270356 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4e3e5f_2493_490f_bbee_79748e604a3f.slice/crio-bc3600f3b687b925e51c6c614a36dae0058570fba290168fe440481174e82c79 WatchSource:0}: Error finding container bc3600f3b687b925e51c6c614a36dae0058570fba290168fe440481174e82c79: Status 404 returned error can't find the container with id bc3600f3b687b925e51c6c614a36dae0058570fba290168fe440481174e82c79 Jul 06 20:14:46.619838 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.619754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:46.619838 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.619792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:46.620078 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.619939 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:46.620078 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.619999 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:46.620078 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.620014 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:14:47.619994279 +0000 UTC m=+4.110617821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:46.620078 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.620038 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:47.620028618 +0000 UTC m=+4.110652154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:46.720192 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.720153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:46.720401 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.720312 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:14:46.720401 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.720336 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:14:46.720401 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.720347 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:46.720570 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:46.720409 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:47.720390257 +0000 UTC m=+4.211013789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:46.968143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.968040 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:09:45 +0000 UTC" deadline="2028-03-11 12:31:06.302625174 +0000 UTC" Jul 06 20:14:46.968143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:46.968091 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14728h16m19.334537019s" Jul 06 20:14:47.077861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.077818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"bc3600f3b687b925e51c6c614a36dae0058570fba290168fe440481174e82c79"} Jul 06 20:14:47.079717 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.079682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlmgh" event={"ID":"38fdc5ad-4828-437f-81fa-8640d7a07e02","Type":"ContainerStarted","Data":"f13ff712808ce99a0721bdccf6e296e71c6b1ea823b80ebd0b652e6f8ebe5f3d"} Jul 06 20:14:47.082143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.082116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrf76" event={"ID":"edf2b52e-ccb9-483b-a378-5de693c9fbc2","Type":"ContainerStarted","Data":"9e029d17cd5a0167c14755ffdf3310b1c1392055203eb056b689559c89ce7561"} Jul 06 20:14:47.084017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.083993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmtzl" event={"ID":"efc56e0e-e63c-4976-b894-0775506ea70b","Type":"ContainerStarted","Data":"ef573119a9e5ca914e6f0df0134919306fc9c090a5ffdcf0ba47d7f441bce94e"} Jul 06 20:14:47.085693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.085674 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4djqt" event={"ID":"e59897e4-965e-4df8-899d-70fcfe6d1ed8","Type":"ContainerStarted","Data":"a9401973c0d522fd3034f7f8b344fa4b940e3d2a34f5a35e589de55b4eeb4a15"} Jul 06 20:14:47.086751 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.086729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerStarted","Data":"4aae19429576eda34fd8e9db263640616635a6fa3b1edc59261df3ab4aa7da61"} Jul 06 20:14:47.089238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.089214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" event={"ID":"3a017b21-d982-4533-bea5-899975060759","Type":"ContainerStarted","Data":"15653faebf0eae1dfcaa2de3c1f32a4f23338e4cea90e5ad2c5898c2e33c757a"} Jul 06 20:14:47.090735 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.090711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5clp9" event={"ID":"7fb82de6-3f26-4671-89d1-4b13446422bf","Type":"ContainerStarted","Data":"e92306c47a07eefc4ff75c9c49d9a2a74eba9ce15f2953a2bf4e638db2e173b1"} Jul 06 20:14:47.626538 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.626493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:47.626538 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.626542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:47.626764 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.626661 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:47.626764 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.626720 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:49.626702825 +0000 UTC m=+6.117326376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:47.626873 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.626780 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:47.626873 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.626812 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:14:49.626801416 +0000 UTC m=+6.117424961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:47.727459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.726856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:47.727459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.727045 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:14:47.727459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.727072 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:14:47.727459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.727086 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:47.727459 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:47.727145 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:49.727127351 +0000 UTC m=+6.217750881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:47.763150 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.763127 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:14:47.968592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.968368 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:09:45 +0000 UTC" deadline="2027-12-30 05:22:33.404774991 +0000 UTC" Jul 06 20:14:47.968592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:47.968593 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12993h7m45.436188473s" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:48.068530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:48.068656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:48.069063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:48.069173 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:48.069234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:48.069411 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:48.069296 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:48.099121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:48.098458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" event={"ID":"5457eba2732788e8cf4686bd84b1d393","Type":"ContainerStarted","Data":"5ca2468fc88e7bb595d85fab34ed6383de10c043627697d5c6b693199026e62e"} Jul 06 20:14:48.114975 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:48.114571 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-104.ec2.internal" podStartSLOduration=3.114553738 podStartE2EDuration="3.114553738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:48.114338567 +0000 UTC m=+4.604962119" watchObservedRunningTime="2026-07-06 20:14:48.114553738 +0000 UTC m=+4.605177292" Jul 06 20:14:49.112774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:49.112739 2574 generic.go:356] "Generic (PLEG): container finished" podID="37165545d2531421ef847dfe3fcc003e" containerID="cce45f2a49b839d3ac57e6f136699d19185e3a1d9c20192737128952300f6a56" exitCode=0 Jul 06 20:14:49.113274 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:49.112845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" event={"ID":"37165545d2531421ef847dfe3fcc003e","Type":"ContainerDied","Data":"cce45f2a49b839d3ac57e6f136699d19185e3a1d9c20192737128952300f6a56"} Jul 06 20:14:49.641430 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:49.641338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:49.641600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:49.641438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:49.641600 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.641497 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:49.641600 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.641536 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:49.641600 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.641573 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:53.641551959 +0000 UTC m=+10.132175489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:49.641600 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.641595 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:14:53.641584109 +0000 UTC m=+10.132207642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:49.742309 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:49.742275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:49.742486 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.742429 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:14:49.742486 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.742448 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:14:49.742486 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.742460 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:49.742633 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:49.742519 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:14:53.742501414 +0000 UTC m=+10.233124942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.065985 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:50.066084 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.066369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:50.066430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.066484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:50.066584 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:50.066539 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:50.118630 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.118022 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" event={"ID":"37165545d2531421ef847dfe3fcc003e","Type":"ContainerStarted","Data":"0d3fce8a501d6f1b332668e1f6d1f5ea3fe7274b04f2cfaa4bcb8c44f80e5886"} Jul 06 20:14:50.121012 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.120587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5clp9" event={"ID":"7fb82de6-3f26-4671-89d1-4b13446422bf","Type":"ContainerStarted","Data":"275d0bf2b9d4bd210f62f55e72d3692841d8e3cd5217b7e385cb3b9d190a1cd0"} Jul 06 20:14:50.140695 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:50.140211 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-104.ec2.internal" podStartSLOduration=5.140169038 podStartE2EDuration="5.140169038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:50.139040714 +0000 UTC m=+6.629664266" watchObservedRunningTime="2026-07-06 20:14:50.140169038 +0000 UTC m=+6.630792592" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.066095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:52.066228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.066588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:52.066693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.066767 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:52.066941 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:52.066852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:52.490507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.490469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:52.491330 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.491304 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:14:52.510820 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:52.510769 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5clp9" podStartSLOduration=5.488375817 podStartE2EDuration="8.510755165s" totalImagesPullingTime="3.022379348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:14:50.160588181 +0000 UTC m=+6.651211731" watchObservedRunningTime="2026-07-06 20:14:52.510755165 +0000 UTC m=+9.001378714" Jul 06 20:14:53.673042 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:53.673008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:53.673509 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:53.673104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:53.673509 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.673173 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:53.673509 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.673212 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:53.673509 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.673243 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:01.673222009 +0000 UTC m=+18.163845544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:14:53.673509 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.673265 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:15:01.673250138 +0000 UTC m=+18.163873679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:14:53.774373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:53.774343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:53.774529 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.774511 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:14:53.774580 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.774536 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:14:53.774580 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.774549 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:53.774643 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:53.774599 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:01.774585171 +0000 UTC m=+18.265208713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:14:54.067290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:54.067258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:54.067472 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:54.067381 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:54.067755 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:54.067738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:54.068023 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:54.067830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:54.068023 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:54.067898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:54.068023 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:54.067988 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:56.066055 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:56.066022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:56.066500 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:56.066150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:56.066500 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:56.066152 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:56.066500 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:56.066199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:56.066500 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:56.066275 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:56.066500 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:56.066350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:58.068821 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.068791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:14:58.069323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.068885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:14:58.069323 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:58.068893 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:14:58.069323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.068917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:14:58.069323 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:58.069005 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:14:58.069323 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:14:58.069105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:14:58.160699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.160655 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:58.268485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.268454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:58.403173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.403104 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-b88jl"] Jul 06 20:14:58.420584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.420555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.423154 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.423126 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:14:58.423255 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.423194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:14:58.423476 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.423455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rq42z\"" Jul 06 20:14:58.443234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.443208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:58.512290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.512266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfb46ee-7922-4f25-8411-0f5c8c6554e0-hosts-file\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.512374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.512357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/edfb46ee-7922-4f25-8411-0f5c8c6554e0-kube-api-access-rqfwj\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.512411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.512393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfb46ee-7922-4f25-8411-0f5c8c6554e0-tmp-dir\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.541855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.541833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:14:58.613299 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.613275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/edfb46ee-7922-4f25-8411-0f5c8c6554e0-kube-api-access-rqfwj\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.613447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.613316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfb46ee-7922-4f25-8411-0f5c8c6554e0-tmp-dir\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.613447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.613366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfb46ee-7922-4f25-8411-0f5c8c6554e0-hosts-file\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.613554 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.613450 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edfb46ee-7922-4f25-8411-0f5c8c6554e0-hosts-file\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.613702 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.613680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfb46ee-7922-4f25-8411-0f5c8c6554e0-tmp-dir\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.624317 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.624290 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfwj\" (UniqueName: \"kubernetes.io/projected/edfb46ee-7922-4f25-8411-0f5c8c6554e0-kube-api-access-rqfwj\") pod \"node-resolver-b88jl\" (UID: \"edfb46ee-7922-4f25-8411-0f5c8c6554e0\") " pod="openshift-dns/node-resolver-b88jl" Jul 06 20:14:58.730163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:14:58.730138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b88jl" Jul 06 20:15:00.066356 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:00.066321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:00.066787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:00.066321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:00.066787 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:00.066429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:00.066787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:00.066322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:00.066787 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:00.066505 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:00.066787 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:00.066603 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:00.539578 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:00.539532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:00.638276 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:00.638242 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:01.737982 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:01.737949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:01.738421 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:01.738010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:01.738421 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.738055 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:15:01.738421 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.738111 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:15:17.738096361 +0000 UTC m=+34.228719906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:15:01.738421 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.738170 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:15:01.738421 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.738231 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:17.73821398 +0000 UTC m=+34.228837521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:15:01.839155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:01.839128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:01.839313 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.839288 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:15:01.839313 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.839306 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:15:01.839404 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.839317 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:15:01.839404 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:01.839361 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:17.839346721 +0000 UTC m=+34.329970249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:15:02.066407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:02.066329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:02.066536 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:02.066469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:02.066536 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:02.066334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:02.066621 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:02.066569 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:02.066728 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:02.066704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:02.066828 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:02.066790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:03.187781 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:03.187749 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:03.289369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:03.289341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:04.066565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:04.066538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:04.066721 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:04.066630 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:04.066761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:04.066719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:04.066833 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:04.066807 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:04.066967 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:04.066864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:04.066967 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:04.066959 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:04.993770 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:04.993738 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfb46ee_7922_4f25_8411_0f5c8c6554e0.slice/crio-e80064258d6ea9fc7723cd3892752b759d708ed4bd1aac814b5d90644531c616 WatchSource:0}: Error finding container e80064258d6ea9fc7723cd3892752b759d708ed4bd1aac814b5d90644531c616: Status 404 returned error can't find the container with id e80064258d6ea9fc7723cd3892752b759d708ed4bd1aac814b5d90644531c616 Jul 06 20:15:05.162932 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.162697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrf76" event={"ID":"edf2b52e-ccb9-483b-a378-5de693c9fbc2","Type":"ContainerStarted","Data":"070c57f73c143db35b23b13c7ea28b57bb99911ed7f91a9ce00bae0a3bb2c222"} Jul 06 20:15:05.164668 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.164589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4djqt" event={"ID":"e59897e4-965e-4df8-899d-70fcfe6d1ed8","Type":"ContainerStarted","Data":"92dd82abcca0d6bc45e3383a791703e2039e1629ef617edd7c8a7c6b75a4989c"} Jul 06 20:15:05.167741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.167229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerStarted","Data":"bf3e812e0d9be86e12005ebd7c05bf83b34df0c3332e59e649cd97df0af519bf"} Jul 06 20:15:05.172000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.171790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b88jl" event={"ID":"edfb46ee-7922-4f25-8411-0f5c8c6554e0","Type":"ContainerStarted","Data":"e80064258d6ea9fc7723cd3892752b759d708ed4bd1aac814b5d90644531c616"} Jul 06 20:15:05.188705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.188655 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrf76" podStartSLOduration=2.578677474 podStartE2EDuration="21.188637305s" totalImagesPullingTime="18.609959831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:05.188096566 +0000 UTC m=+21.678720117" watchObservedRunningTime="2026-07-06 20:15:05.188637305 +0000 UTC m=+21.679260860" Jul 06 20:15:05.233018 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:05.232976 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4djqt" podStartSLOduration=6.875823527 podStartE2EDuration="21.232962124s" totalImagesPullingTime="14.357138597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:05.232620441 +0000 UTC m=+21.723243985" watchObservedRunningTime="2026-07-06 20:15:05.232962124 +0000 UTC m=+21.723585679" Jul 06 20:15:06.066528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.066359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:06.067294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.066357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:06.067294 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:06.066631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:06.067294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.066406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:06.067294 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:06.066663 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:06.067294 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:06.066741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:06.175632 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.175605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" event={"ID":"3a017b21-d982-4533-bea5-899975060759","Type":"ContainerStarted","Data":"4ef78e9acd4a4deed3f16339d9963725f68975f84b7c1a23d52fa5dcec4d994f"} Jul 06 20:15:06.177017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.176985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b88jl" event={"ID":"edfb46ee-7922-4f25-8411-0f5c8c6554e0","Type":"ContainerStarted","Data":"79b8c57fc43b04a28607ea24f29798e28a254f0b42ac8a9048bdc3b023e37b77"} Jul 06 20:15:06.179571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.179555 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:15:06.179888 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.179866 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a4e3e5f-2493-490f-bbee-79748e604a3f" containerID="c9c1afc28e54915921ab7d7cc533fc0328595b4d08a4f6507954a83e34f604f6" exitCode=1 Jul 06 20:15:06.179965 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.179926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"b25cac0d63fbf346a1eb2d363716d0ccceb2877ac9aab44c8297ce6e7a311851"} Jul 06 20:15:06.180018 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.180007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"fbd67de54e230052abf676647a5ef1c634d697786ad75b4425052d39cfdca04e"} Jul 06 20:15:06.180071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.180058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"14ef02e9cd204ab225e0c17bbbdd5c9b8924700a782d5fcf607e853439249f6b"} Jul 06 20:15:06.180112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.180099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"5c7b1d81d70ee4f68791b6c65c9a0bb8899a78ecef6588c17ecc0deb5bb94319"} Jul 06 20:15:06.180169 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.180145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerDied","Data":"c9c1afc28e54915921ab7d7cc533fc0328595b4d08a4f6507954a83e34f604f6"} Jul 06 20:15:06.180218 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.180176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"21a79aed0e1496a56735f7b8e6f97d63afa36b6c63e2c43c242f2a9e1a3b0154"} Jul 06 20:15:06.181273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.181255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qlmgh" event={"ID":"38fdc5ad-4828-437f-81fa-8640d7a07e02","Type":"ContainerStarted","Data":"8ca890b9ef83770128f227e83114f088bd8e89d656be8ae9566025016f9d4b7a"} Jul 06 20:15:06.182464 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.182443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmtzl" event={"ID":"efc56e0e-e63c-4976-b894-0775506ea70b","Type":"ContainerStarted","Data":"979aa0fd3f7faeac0afa2aa817c843a13aafd831e0617ceea6b2dc0af7f12f87"} Jul 06 20:15:06.183679 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.183660 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="bf3e812e0d9be86e12005ebd7c05bf83b34df0c3332e59e649cd97df0af519bf" exitCode=0 Jul 06 20:15:06.183968 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.183940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"bf3e812e0d9be86e12005ebd7c05bf83b34df0c3332e59e649cd97df0af519bf"} Jul 06 20:15:06.194261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.194227 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b88jl" podStartSLOduration=8.19421702 podStartE2EDuration="8.19421702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:06.193890966 +0000 UTC m=+22.684514517" watchObservedRunningTime="2026-07-06 20:15:06.19421702 +0000 UTC m=+22.684840569" Jul 06 20:15:06.214215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.214181 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cmtzl" podStartSLOduration=3.471676535 podStartE2EDuration="22.214170576s" totalImagesPullingTime="18.742494041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:06.213853876 +0000 UTC m=+22.704477428" watchObservedRunningTime="2026-07-06 20:15:06.214170576 +0000 UTC m=+22.704794126" Jul 06 20:15:06.236297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.236261 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qlmgh" podStartSLOduration=7.905276768 podStartE2EDuration="22.236248695s" totalImagesPullingTime="14.330971927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:06.235764391 +0000 UTC m=+22.726387940" watchObservedRunningTime="2026-07-06 20:15:06.236248695 +0000 UTC m=+22.726872244" Jul 06 20:15:06.385433 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.385360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:06.487808 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.487767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:06.656379 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.656348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:06.754974 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.754949 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:06.834069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.834049 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:15:06.968857 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.968701 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:15:06.834064833Z","UUID":"e84c5a9d-7b9d-4a38-beea-f0ab7a712519","Handler":null,"Name":"","Endpoint":""} Jul 06 20:15:06.970245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.970223 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:15:06.970503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:06.970261 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:15:07.187294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:07.187260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" event={"ID":"3a017b21-d982-4533-bea5-899975060759","Type":"ContainerStarted","Data":"0d50eeb7cd3573b33114194e0445b5f57fe768aac4722e09481deb5b6cd0d882"} Jul 06 20:15:08.059683 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.059652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:08.066736 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.066622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:08.066858 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:08.066734 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:08.067115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.067096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:08.067253 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:08.067208 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:08.067352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.067337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:08.067435 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:08.067415 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:08.158872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.158847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:08.193791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.193772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:15:08.194215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.194186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"09846490d4449b35220f13695db162c93723613ecb27690cb00c5533a9cb9eae"} Jul 06 20:15:08.196863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.196833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" event={"ID":"3a017b21-d982-4533-bea5-899975060759","Type":"ContainerStarted","Data":"07421e967f8c8b13ed5980ec5619cea07871664858e79c4a319e14960204e609"} Jul 06 20:15:08.211205 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.211182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:15:08.211308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.211287 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 20:15:08.211815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.211778 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5clp9" Jul 06 20:15:08.244144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.244106 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-76tlg" podStartSLOduration=3.001820578 podStartE2EDuration="24.244092617s" totalImagesPullingTime="21.242272039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:08.221031479 +0000 UTC m=+24.711655030" watchObservedRunningTime="2026-07-06 20:15:08.244092617 +0000 UTC m=+24.734716218" Jul 06 20:15:08.315304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.315142 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:08.967841 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:08.967808 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:09.063691 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:09.063663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:09.161234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:09.161204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:09.729612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:09.729573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:09.818767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:09.818745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:10.066350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.066324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:10.066350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.066336 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:10.066538 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.066329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:10.066538 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:10.066418 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:10.066538 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:10.066511 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:10.066690 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:10.066566 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:10.205084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.205013 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:15:10.205558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.205475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"c001cbe22475d23d2b29392ac3f4479bd7b3edad8cbc7b3aa257808502f308f7"} Jul 06 20:15:10.205889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.205869 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:10.206012 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.205962 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:10.206428 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.206084 2574 scope.go:117] "RemoveContainer" containerID="c9c1afc28e54915921ab7d7cc533fc0328595b4d08a4f6507954a83e34f604f6" Jul 06 20:15:10.223979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.223955 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:10.224648 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.224608 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:10.565149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.565072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:10.644647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:10.644611 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.072117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.072081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.157125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.157091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.211600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.211577 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:15:11.212014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.211988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" event={"ID":"1a4e3e5f-2493-490f-bbee-79748e604a3f","Type":"ContainerStarted","Data":"7a1722a94248e8c62ce1d5353600e4b4590920bc1f4ffbb2a22329f2da57d2a3"} Jul 06 20:15:11.212136 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.212088 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 20:15:11.244389 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.244347 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" podStartSLOduration=8.360620401 podStartE2EDuration="27.244335947s" totalImagesPullingTime="18.883715546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:11.244183313 +0000 UTC m=+27.734806862" watchObservedRunningTime="2026-07-06 20:15:11.244335947 +0000 UTC m=+27.734959497" Jul 06 20:15:11.423219 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.423183 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.507209 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.507176 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.614412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.614379 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:11.714263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:11.714236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:12.066675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.066615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:12.066847 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.066627 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:12.066847 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.066717 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:12.066847 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.066732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:12.066847 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.066821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:12.067046 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.066941 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:12.238043 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.238024 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 20:15:12.453995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.453965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hbp9"] Jul 06 20:15:12.454082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.454067 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:12.454163 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.454145 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:12.456704 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.456685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mpq6z"] Jul 06 20:15:12.456799 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.456752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:12.456836 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.456813 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:12.460711 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.460693 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d9crx"] Jul 06 20:15:12.460777 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:12.460761 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:12.460843 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:12.460829 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:13.021794 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.021769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:13.218231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.218200 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="33f0721831d92fa6f236bf7316354823dadaf72e81a9c24af9ab01d354ef7872" exitCode=0 Jul 06 20:15:13.218376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.218232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"33f0721831d92fa6f236bf7316354823dadaf72e81a9c24af9ab01d354ef7872"} Jul 06 20:15:13.294649 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.294588 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:13.567105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.567058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:13.603714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.603678 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:13.708157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.708122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:13.866334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.866275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:13.968393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:13.968350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:14.067324 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:14.067296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:14.067471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:14.067396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:14.067471 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:14.067412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:14.067573 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:14.067500 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:14.067573 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:14.067541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:14.067687 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:14.067666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:14.963170 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:14.963138 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:15.157859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:15.157828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:15.224681 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:15.224618 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="b6f6f211714df72df1e8472926004562a3d43810fd58c35c4e4adb11c716c8a0" exitCode=0 Jul 06 20:15:15.224681 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:15.224654 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"b6f6f211714df72df1e8472926004562a3d43810fd58c35c4e4adb11c716c8a0"} Jul 06 20:15:15.362426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:15.362405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:15.809887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:15.809863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:16.065968 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.065885 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:16.066476 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.065886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:16.066476 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:16.066035 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:16.066476 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:16.066099 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:16.066476 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.065895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:16.066476 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:16.066183 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:16.574289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.574262 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:16.766418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.766391 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:16.868017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.867965 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:16.964688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:16.964666 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.170244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.170211 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.229781 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.229753 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="5f9c4435b5f9f6963809e1f594cdada071faa30e613c24550a09d882f23ce79d" exitCode=0 Jul 06 20:15:17.229916 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.229811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"5f9c4435b5f9f6963809e1f594cdada071faa30e613c24550a09d882f23ce79d"} Jul 06 20:15:17.267243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.267214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.367718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.367698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.592813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.592761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.702439 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.702419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:17.753120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.753102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:17.753217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.753150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:17.753258 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.753228 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:15:17.753258 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.753231 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:15:17.753321 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.753270 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs podName:de5a9039-dd79-406d-96c8-0e10737ecf6d nodeName:}" failed. No retries permitted until 2026-07-06 20:15:49.753257514 +0000 UTC m=+66.243881041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs") pod "network-metrics-daemon-d9crx" (UID: "de5a9039-dd79-406d-96c8-0e10737ecf6d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:15:17.753321 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.753282 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret podName:5fd1d381-faee-4d65-9d12-05c0a91abc96 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:49.7532755 +0000 UTC m=+66.243899028 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret") pod "global-pull-secret-syncer-mpq6z" (UID: "5fd1d381-faee-4d65-9d12-05c0a91abc96") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:15:17.853560 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:17.853518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:17.853672 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.853627 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:15:17.853672 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.853638 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:15:17.853672 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.853646 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6sfl9 for pod openshift-network-diagnostics/network-check-target-5hbp9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:15:17.853767 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:17.853682 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9 podName:3036029f-9306-4f30-a02e-b0d5f00e1cd4 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:49.853670631 +0000 UTC m=+66.344294159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sfl9" (UniqueName: "kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9") pod "network-check-target-5hbp9" (UID: "3036029f-9306-4f30-a02e-b0d5f00e1cd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:15:18.066452 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:18.066428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:18.066600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:18.066493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:18.066600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:18.066559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:18.066716 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:18.066602 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:18.066716 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:18.066675 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:18.066821 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:18.066790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:18.568878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:18.568846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:18.771019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:18.770980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:19.313103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:19.313075 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:19.453431 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:19.453403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:19.576720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:19.576644 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:19.826875 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:19.826795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:19.930215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:19.930189 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:20.043917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.043881 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:20.065999 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.065976 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:20.066110 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.065977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:20.066110 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:20.066098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:20.066225 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.066113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:20.066281 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:20.066215 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:20.066339 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:20.066295 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:20.587158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.587131 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:20.807424 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.807389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:20.987616 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:20.987583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:21.081892 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:21.081865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:21.215882 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:21.215841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:21.530621 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:21.530593 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:21.807745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:21.807676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:22.066577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.066509 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:22.066577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.066518 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:22.066577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.066529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:22.066826 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:22.066603 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:22.066826 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:22.066656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:22.066826 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:22.066709 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:22.369558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.369463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:22.771355 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.771330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:22.955353 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:22.955324 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:23.136657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:23.136538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:23.206099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:23.206069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:23.340415 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:23.340378 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:23.742567 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:23.742535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:23.947777 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:23.947740 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:24.066643 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:24.066571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:24.067117 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:24.066682 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:24.067117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:24.066786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:24.067117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:24.066838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:24.067117 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:24.066946 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:24.067117 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:24.066975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:25.248246 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:25.248217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerStarted","Data":"b3e8f8bcecb9e5857b04719fe1c501014ccbcfa7f69faad0b57688f1ce46b7df"} Jul 06 20:15:25.431207 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:25.431179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:26.066065 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:26.066041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:26.066226 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:26.066066 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:26.066226 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:26.066044 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:26.066226 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:26.066155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:26.066392 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:26.066239 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:26.066392 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:26.066329 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:26.252866 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:26.252842 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="b3e8f8bcecb9e5857b04719fe1c501014ccbcfa7f69faad0b57688f1ce46b7df" exitCode=0 Jul 06 20:15:26.253174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:26.252875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"b3e8f8bcecb9e5857b04719fe1c501014ccbcfa7f69faad0b57688f1ce46b7df"} Jul 06 20:15:27.257535 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:27.257496 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b7ec425-fa4f-4648-8e08-561cb0cb7984" containerID="bc44406ccb6be0735c991fb1f4a421b22bf69d13ec545f6af8ef53b2773f1fe7" exitCode=0 Jul 06 20:15:27.257880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:27.257550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerDied","Data":"bc44406ccb6be0735c991fb1f4a421b22bf69d13ec545f6af8ef53b2773f1fe7"} Jul 06 20:15:28.066239 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:28.066209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:28.066391 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:28.066216 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:28.066391 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:28.066301 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:28.066391 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:28.066370 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:28.066391 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:28.066218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:28.066515 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:28.066472 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:28.263027 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:28.263004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" event={"ID":"9b7ec425-fa4f-4648-8e08-561cb0cb7984","Type":"ContainerStarted","Data":"dfbbdf2d34272c76c192a32f9d4852924118586abd85eb86b4eff4fe490b4fbc"} Jul 06 20:15:28.287932 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:28.287877 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rm2rx" podStartSLOduration=9.250525974 podStartE2EDuration="44.287865096s" totalImagesPullingTime="35.037339122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:28.287503336 +0000 UTC m=+44.778126885" watchObservedRunningTime="2026-07-06 20:15:28.287865096 +0000 UTC m=+44.778488624" Jul 06 20:15:29.440888 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:29.440859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:29.508059 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:29.508034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:30.065614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:30.065583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:30.065799 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:30.065584 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:30.065799 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:30.065705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:30.065799 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:30.065703 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:30.065799 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:30.065775 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:30.066033 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:30.065865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:30.517380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:30.517350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:30.572678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:30.572654 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:32.065765 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:32.065739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:32.066133 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:32.065740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:32.066133 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:32.065836 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:32.066133 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:32.065745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:32.066133 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:32.065917 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:32.066133 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:32.066003 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:33.518122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:33.518091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:33.672887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:33.672859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:33.782443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:33.782384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:33.883843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:33.883821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:33.978661 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:33.978638 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:34.067445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.067388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:34.067536 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.067491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:34.067536 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:34.067504 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mpq6z" podUID="5fd1d381-faee-4d65-9d12-05c0a91abc96" Jul 06 20:15:34.067647 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:34.067559 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hbp9" podUID="3036029f-9306-4f30-a02e-b0d5f00e1cd4" Jul 06 20:15:34.067647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.067564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:34.067745 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:34.067659 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d9crx" podUID="de5a9039-dd79-406d-96c8-0e10737ecf6d" Jul 06 20:15:34.608815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.608789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:34.669301 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.669281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:34.757094 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.757042 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-104.ec2.internal" event="NodeReady" Jul 06 20:15:34.757197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.757152 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:15:34.794938 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.794895 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c9b96759f-dmn9j"] Jul 06 20:15:34.796680 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.796664 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.799253 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.799230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xfxxg\"" Jul 06 20:15:34.799360 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.799230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:15:34.799360 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.799288 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:15:34.799360 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.799303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:15:34.804495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.804473 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:15:34.822779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.822761 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:15:34.828054 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.828035 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9b96759f-dmn9j"] Jul 06 20:15:34.828157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.828119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.828350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.828329 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rbhmr"] Jul 06 20:15:34.830091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830075 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pxw7m"] Jul 06 20:15:34.830194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.830624 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830598 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 20:15:34.830714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:15:34.830714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830671 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:15:34.830714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830678 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:15:34.830878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830784 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lltw8\"" Jul 06 20:15:34.830947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830926 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:15:34.830947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830944 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 20:15:34.831048 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.830954 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:15:34.831973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.831958 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.833702 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.833685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9vhmq\"" Jul 06 20:15:34.833959 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.833946 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:15:34.834031 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.834022 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:15:34.834119 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.834104 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:15:34.834281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.834241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk74q\"" Jul 06 20:15:34.834408 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.834320 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:15:34.834643 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.834624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:15:34.838125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.838093 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbhmr"] Jul 06 20:15:34.838669 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.838651 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:15:34.843438 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.843419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pxw7m"] Jul 06 20:15:34.881525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-bound-sa-token\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.881627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13edad41-5e3f-4825-a0ce-9004a505e33f-config-volume\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.881627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13edad41-5e3f-4825-a0ce-9004a505e33f-metrics-tls\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.881627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-certificates\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.881765 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.881765 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce5bf723-805b-415f-b2c1-ce704b1b1b65-ca-trust-extracted\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.881765 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-installation-pull-secrets\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.881934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.881934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.881934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edad41-5e3f-4825-a0ce-9004a505e33f-tmp-dir\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.881934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-cert\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.881934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-image-registry-private-configuration\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.881998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-tls\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-kube-api-access-5f55n\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cts6n\" (UniqueName: \"kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882093 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztrm\" (UniqueName: \"kubernetes.io/projected/13edad41-5e3f-4825-a0ce-9004a505e33f-kube-api-access-dztrm\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgnc\" (UniqueName: \"kubernetes.io/projected/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-kube-api-access-6wgnc\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.882125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.882121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-trusted-ca\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgnc\" (UniqueName: \"kubernetes.io/projected/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-kube-api-access-6wgnc\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.982607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-trusted-ca\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-bound-sa-token\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13edad41-5e3f-4825-a0ce-9004a505e33f-config-volume\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.982718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13edad41-5e3f-4825-a0ce-9004a505e33f-metrics-tls\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.982718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-certificates\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.982718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce5bf723-805b-415f-b2c1-ce704b1b1b65-ca-trust-extracted\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-installation-pull-secrets\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.982891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.982777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edad41-5e3f-4825-a0ce-9004a505e33f-tmp-dir\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-cert\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983230 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce5bf723-805b-415f-b2c1-ce704b1b1b65-ca-trust-extracted\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-image-registry-private-configuration\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-tls\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-kube-api-access-5f55n\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cts6n\" (UniqueName: \"kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.983537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztrm\" (UniqueName: \"kubernetes.io/projected/13edad41-5e3f-4825-a0ce-9004a505e33f-kube-api-access-dztrm\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.984160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-certificates\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.984160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983653 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce5bf723-805b-415f-b2c1-ce704b1b1b65-trusted-ca\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.984160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.984160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983707 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.984160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.983875 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13edad41-5e3f-4825-a0ce-9004a505e33f-tmp-dir\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.984650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.984610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.984650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.984625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13edad41-5e3f-4825-a0ce-9004a505e33f-config-volume\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.987660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987516 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-installation-pull-secrets\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.987660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13edad41-5e3f-4825-a0ce-9004a505e33f-metrics-tls\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.987660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987604 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.987660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987618 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:34.987660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-cert\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.987892 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.987704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-registry-tls\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.988029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.988009 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce5bf723-805b-415f-b2c1-ce704b1b1b65-image-registry-private-configuration\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.992381 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.992363 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-bound-sa-token\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.992494 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.992441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgnc\" (UniqueName: \"kubernetes.io/projected/e0d27bad-d36b-409c-afdd-8bc6ac3a8e91-kube-api-access-6wgnc\") pod \"ingress-canary-rbhmr\" (UID: \"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91\") " pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:34.992642 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.992616 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztrm\" (UniqueName: \"kubernetes.io/projected/13edad41-5e3f-4825-a0ce-9004a505e33f-kube-api-access-dztrm\") pod \"dns-default-pxw7m\" (UID: \"13edad41-5e3f-4825-a0ce-9004a505e33f\") " pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:34.992744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.992727 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/ce5bf723-805b-415f-b2c1-ce704b1b1b65-kube-api-access-5f55n\") pod \"image-registry-6c9b96759f-dmn9j\" (UID: \"ce5bf723-805b-415f-b2c1-ce704b1b1b65\") " pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:34.993924 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:34.993894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cts6n\" (UniqueName: \"kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n\") pod \"console-559d66d65b-5g2g6\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:35.106044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.106018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:35.139922 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.139045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:35.147005 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.146978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbhmr" Jul 06 20:15:35.151322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.151173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:35.312295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.312263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:15:35.316155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.316132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9b96759f-dmn9j"] Jul 06 20:15:35.316579 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:35.316555 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda534f7b6_75ce_40a3_ad2d_fdedc2044eba.slice/crio-120a556b26f51f6860fe62a6c80192d7b53c2052324f71ebe41b64d9e8c53223 WatchSource:0}: Error finding container 120a556b26f51f6860fe62a6c80192d7b53c2052324f71ebe41b64d9e8c53223: Status 404 returned error can't find the container with id 120a556b26f51f6860fe62a6c80192d7b53c2052324f71ebe41b64d9e8c53223 Jul 06 20:15:35.320462 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:35.320441 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5bf723_805b_415f_b2c1_ce704b1b1b65.slice/crio-4cccff990ab2cde7d8ea2afc99ddd76fe8242ceca71021a5cf9276690927de87 WatchSource:0}: Error finding container 4cccff990ab2cde7d8ea2afc99ddd76fe8242ceca71021a5cf9276690927de87: Status 404 returned error can't find the container with id 4cccff990ab2cde7d8ea2afc99ddd76fe8242ceca71021a5cf9276690927de87 Jul 06 20:15:35.520220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.520183 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbhmr"] Jul 06 20:15:35.522913 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:35.522855 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pxw7m"] Jul 06 20:15:35.524358 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:35.524332 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d27bad_d36b_409c_afdd_8bc6ac3a8e91.slice/crio-0fac03c7eac76b799d15ac68f852d2bd9df10113d05d4e2c03c87021f19e1362 WatchSource:0}: Error finding container 0fac03c7eac76b799d15ac68f852d2bd9df10113d05d4e2c03c87021f19e1362: Status 404 returned error can't find the container with id 0fac03c7eac76b799d15ac68f852d2bd9df10113d05d4e2c03c87021f19e1362 Jul 06 20:15:35.525946 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:35.525920 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13edad41_5e3f_4825_a0ce_9004a505e33f.slice/crio-3604b416b5c0391f90d5c339fa53baa17838b275c3aa7a90d548b7d8a6127c36 WatchSource:0}: Error finding container 3604b416b5c0391f90d5c339fa53baa17838b275c3aa7a90d548b7d8a6127c36: Status 404 returned error can't find the container with id 3604b416b5c0391f90d5c339fa53baa17838b275c3aa7a90d548b7d8a6127c36 Jul 06 20:15:36.066592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.066490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:36.067076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.066490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:36.067076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.066923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:36.069542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.069520 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:15:36.071176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.071008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ksdkk\"" Jul 06 20:15:36.071176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.071021 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:15:36.071176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.071106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8g8p6\"" Jul 06 20:15:36.071176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.071106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:15:36.071471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.071106 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:15:36.282595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.282561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" event={"ID":"ce5bf723-805b-415f-b2c1-ce704b1b1b65","Type":"ContainerStarted","Data":"180669d54c228afb5f540842cf5c7bf9ca3fcb4f3fccef178b367e3da246f892"} Jul 06 20:15:36.282740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.282664 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:36.282740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.282689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" event={"ID":"ce5bf723-805b-415f-b2c1-ce704b1b1b65","Type":"ContainerStarted","Data":"4cccff990ab2cde7d8ea2afc99ddd76fe8242ceca71021a5cf9276690927de87"} Jul 06 20:15:36.283917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.283872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pxw7m" event={"ID":"13edad41-5e3f-4825-a0ce-9004a505e33f","Type":"ContainerStarted","Data":"3604b416b5c0391f90d5c339fa53baa17838b275c3aa7a90d548b7d8a6127c36"} Jul 06 20:15:36.285310 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.285265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559d66d65b-5g2g6" event={"ID":"a534f7b6-75ce-40a3-ad2d-fdedc2044eba","Type":"ContainerStarted","Data":"120a556b26f51f6860fe62a6c80192d7b53c2052324f71ebe41b64d9e8c53223"} Jul 06 20:15:36.286890 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.286818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbhmr" event={"ID":"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91","Type":"ContainerStarted","Data":"0fac03c7eac76b799d15ac68f852d2bd9df10113d05d4e2c03c87021f19e1362"} Jul 06 20:15:36.307159 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.307115 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" podStartSLOduration=3.307102001 podStartE2EDuration="3.307102001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:36.306285038 +0000 UTC m=+52.796908588" watchObservedRunningTime="2026-07-06 20:15:36.307102001 +0000 UTC m=+52.797725550" Jul 06 20:15:36.404756 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.404695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:36.505541 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:36.505516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:38.404026 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.403993 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:15:38.407050 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.407030 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.414601 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.414068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:38.416449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.416432 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:15:38.422478 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.422424 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:15:38.484176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.484149 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:38.508597 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h765n\" (UniqueName: \"kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.508885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.508793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610037 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.609999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h765n\" (UniqueName: \"kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.610426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.610287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.611171 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.611138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.611350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.611299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.611473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.611380 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.611785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.611764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.613284 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.613260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.613370 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.613326 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.620754 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.620735 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h765n\" (UniqueName: \"kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n\") pod \"console-56776464-p45nk\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:38.717952 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:38.717918 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:39.164819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:39.164787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:39.673595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:39.673526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:15:39.679556 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:39.679512 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457f825f_ce6f_4f32_a754_fc1d4ddfa3b6.slice/crio-c2c088f0a3a604d8f594ad3a5d29c44eb000d2ecb892f225dfef127e9c5d187e WatchSource:0}: Error finding container c2c088f0a3a604d8f594ad3a5d29c44eb000d2ecb892f225dfef127e9c5d187e: Status 404 returned error can't find the container with id c2c088f0a3a604d8f594ad3a5d29c44eb000d2ecb892f225dfef127e9c5d187e Jul 06 20:15:40.299797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.299761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56776464-p45nk" event={"ID":"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6","Type":"ContainerStarted","Data":"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155"} Jul 06 20:15:40.300009 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.299830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56776464-p45nk" event={"ID":"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6","Type":"ContainerStarted","Data":"c2c088f0a3a604d8f594ad3a5d29c44eb000d2ecb892f225dfef127e9c5d187e"} Jul 06 20:15:40.301630 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.301599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pxw7m" event={"ID":"13edad41-5e3f-4825-a0ce-9004a505e33f","Type":"ContainerStarted","Data":"d16c0e5f1f05d4f531e0b2898534b250edb4334d36cac6e2b3aa9c9a9bdf4ffa"} Jul 06 20:15:40.301761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.301657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pxw7m" event={"ID":"13edad41-5e3f-4825-a0ce-9004a505e33f","Type":"ContainerStarted","Data":"439ad0ce874fb364bc046c6c817fe1606fe5233cd65840ba69f2a4c14dda600f"} Jul 06 20:15:40.301761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.301721 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:40.303093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.303070 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559d66d65b-5g2g6" event={"ID":"a534f7b6-75ce-40a3-ad2d-fdedc2044eba","Type":"ContainerStarted","Data":"bf9c59ef9a4a1aca3522531df437e900c4dea176723939979452858677978ef9"} Jul 06 20:15:40.304407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.304386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbhmr" event={"ID":"e0d27bad-d36b-409c-afdd-8bc6ac3a8e91","Type":"ContainerStarted","Data":"f51b288018efd181036789b340b79dc7b3d48b8c92950f4c3d49079883ee742b"} Jul 06 20:15:40.317018 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.316983 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56776464-p45nk" podStartSLOduration=2.316970002 podStartE2EDuration="2.316970002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:40.316514357 +0000 UTC m=+56.807137906" watchObservedRunningTime="2026-07-06 20:15:40.316970002 +0000 UTC m=+56.807593551" Jul 06 20:15:40.330525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.330489 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rbhmr" podStartSLOduration=2.314244162 podStartE2EDuration="6.330477115s" totalImagesPullingTime="4.016232953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:40.329853563 +0000 UTC m=+56.820477114" watchObservedRunningTime="2026-07-06 20:15:40.330477115 +0000 UTC m=+56.821100664" Jul 06 20:15:40.345058 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.345022 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pxw7m" podStartSLOduration=2.338722553 podStartE2EDuration="6.345009754s" totalImagesPullingTime="4.006287201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:40.344635633 +0000 UTC m=+56.835259182" watchObservedRunningTime="2026-07-06 20:15:40.345009754 +0000 UTC m=+56.835633282" Jul 06 20:15:40.363778 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:40.363744 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559d66d65b-5g2g6" podStartSLOduration=2.142844685 podStartE2EDuration="6.363733725s" totalImagesPullingTime="4.22088904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:40.363316639 +0000 UTC m=+56.853940190" watchObservedRunningTime="2026-07-06 20:15:40.363733725 +0000 UTC m=+56.854357276" Jul 06 20:15:42.399064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:42.399030 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:42.499468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:42.499440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:43.290533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.290503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:43.352988 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.352957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:43.588254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.588206 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6vmlz" Jul 06 20:15:43.750573 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.750553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:43.843897 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.843832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:43.944744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:43.944720 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:44.051227 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.051200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:44.186380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.186352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:44.284824 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.284798 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-glwkf"] Jul 06 20:15:44.318213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.318191 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glwkf"] Jul 06 20:15:44.318363 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.318283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.320823 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.320791 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:15:44.320823 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.320791 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:15:44.320971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.320824 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:15:44.321789 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.321767 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xjp7h\"" Jul 06 20:15:44.321868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.321775 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:15:44.340533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.340507 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:44.355608 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.355585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8be60e6-c948-4081-a1ea-e1fe0012e849-crio-socket\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.355687 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.355617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8be60e6-c948-4081-a1ea-e1fe0012e849-data-volume\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.355687 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.355640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6sg\" (UniqueName: \"kubernetes.io/projected/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-api-access-ld6sg\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.355797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.355708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.355797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.355766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457009 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.456963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8be60e6-c948-4081-a1ea-e1fe0012e849-crio-socket\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457009 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.456992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8be60e6-c948-4081-a1ea-e1fe0012e849-data-volume\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457009 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.457008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6sg\" (UniqueName: \"kubernetes.io/projected/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-api-access-ld6sg\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.457049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.457081 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c8be60e6-c948-4081-a1ea-e1fe0012e849-crio-socket\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.457100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.457198 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.457168 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:15:44.457198 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.457196 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:15:44.457422 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.457236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls podName:c8be60e6-c948-4081-a1ea-e1fe0012e849 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:44.957218265 +0000 UTC m=+61.447841801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glwkf" (UID: "c8be60e6-c948-4081-a1ea-e1fe0012e849") : secret "insights-runtime-extractor-tls" not found Jul 06 20:15:44.457422 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.457253 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm podName:c8be60e6-c948-4081-a1ea-e1fe0012e849 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:44.957244508 +0000 UTC m=+61.447868040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glwkf" (UID: "c8be60e6-c948-4081-a1ea-e1fe0012e849") : configmap "kube-rbac-proxy" not found Jul 06 20:15:44.457422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.457349 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c8be60e6-c948-4081-a1ea-e1fe0012e849-data-volume\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.464998 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.464981 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6sg\" (UniqueName: \"kubernetes.io/projected/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-api-access-ld6sg\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.482024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.482004 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:44.960098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.960080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.960401 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.960122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:44.960401 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.960223 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:15:44.960401 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:44.960280 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls podName:c8be60e6-c948-4081-a1ea-e1fe0012e849 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:45.960263434 +0000 UTC m=+62.450886974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glwkf" (UID: "c8be60e6-c948-4081-a1ea-e1fe0012e849") : secret "insights-runtime-extractor-tls" not found Jul 06 20:15:44.960645 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:44.960630 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c8be60e6-c948-4081-a1ea-e1fe0012e849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:45.003760 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.003736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:45.079752 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.079730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:45.141460 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.141433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:45.141570 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.141481 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:45.142516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.142491 2574 patch_prober.go:28] interesting pod/console-559d66d65b-5g2g6 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.7:8443/health\": dial tcp 10.133.0.7:8443: connect: connection refused" start-of-body= Jul 06 20:15:45.142600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.142532 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-559d66d65b-5g2g6" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerName="console" probeResult="failure" output="Get \"https://10.133.0.7:8443/health\": dial tcp 10.133.0.7:8443: connect: connection refused" Jul 06 20:15:45.144225 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.144203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:45.516231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.516195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:45.576863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.576837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:45.967885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.967846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:45.970046 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.970024 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c8be60e6-c948-4081-a1ea-e1fe0012e849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glwkf\" (UID: \"c8be60e6-c948-4081-a1ea-e1fe0012e849\") " pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:45.994816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:45.994789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:46.097238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.097210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:46.127111 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.127099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glwkf" Jul 06 20:15:46.250875 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.250813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glwkf"] Jul 06 20:15:46.253763 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:46.253738 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8be60e6_c948_4081_a1ea_e1fe0012e849.slice/crio-fd632cb30adfe32f046eaf4b5fdf772d4231728cd14a063ec8f20eca15691167 WatchSource:0}: Error finding container fd632cb30adfe32f046eaf4b5fdf772d4231728cd14a063ec8f20eca15691167: Status 404 returned error can't find the container with id fd632cb30adfe32f046eaf4b5fdf772d4231728cd14a063ec8f20eca15691167 Jul 06 20:15:46.325654 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.325619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glwkf" event={"ID":"c8be60e6-c948-4081-a1ea-e1fe0012e849","Type":"ContainerStarted","Data":"fd632cb30adfe32f046eaf4b5fdf772d4231728cd14a063ec8f20eca15691167"} Jul 06 20:15:46.699888 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.699857 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-559d66d65b-5g2g6_a534f7b6-75ce-40a3-ad2d-fdedc2044eba/console/0.log" Jul 06 20:15:46.900696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:46.900666 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56776464-p45nk_457f825f-ce6f-4f32-a754-fc1d4ddfa3b6/console/0.log" Jul 06 20:15:47.011653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.011533 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.075021 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.074989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.178384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.178349 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.260937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.260897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.299535 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.299472 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pxw7m_13edad41-5e3f-4825-a0ce-9004a505e33f/dns/0.log" Jul 06 20:15:47.499512 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.499486 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pxw7m_13edad41-5e3f-4825-a0ce-9004a505e33f/kube-rbac-proxy/0.log" Jul 06 20:15:47.621936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.621859 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.689395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.689365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.757606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.757577 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:47.906403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:47.906370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:48.700267 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:48.700236 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b88jl_edfb46ee-7922-4f25-8411-0f5c8c6554e0/dns-node-resolver/0.log" Jul 06 20:15:48.718396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:48.718370 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:48.718528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:48.718423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:48.719895 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:48.719861 2574 patch_prober.go:28] interesting pod/console-56776464-p45nk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body= Jul 06 20:15:48.720013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:48.719929 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-56776464-p45nk" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerName="console" probeResult="failure" output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" Jul 06 20:15:49.101039 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.100950 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c9b96759f-dmn9j_ce5bf723-805b-415f-b2c1-ce704b1b1b65/registry/0.log" Jul 06 20:15:49.499809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.499783 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4djqt_e59897e4-965e-4df8-899d-70fcfe6d1ed8/node-ca/0.log" Jul 06 20:15:49.797394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.797322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:49.797394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.797376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:49.799826 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.799810 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:15:49.799920 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.799816 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:15:49.810300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.810270 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5fd1d381-faee-4d65-9d12-05c0a91abc96-original-pull-secret\") pod \"global-pull-secret-syncer-mpq6z\" (UID: \"5fd1d381-faee-4d65-9d12-05c0a91abc96\") " pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:49.810300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.810293 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a9039-dd79-406d-96c8-0e10737ecf6d-metrics-certs\") pod \"network-metrics-daemon-d9crx\" (UID: \"de5a9039-dd79-406d-96c8-0e10737ecf6d\") " pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:49.891832 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.891813 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ksdkk\"" Jul 06 20:15:49.896564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.896544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mpq6z" Jul 06 20:15:49.897763 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.897741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:49.899710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.899688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d9crx" Jul 06 20:15:49.900093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.900076 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:15:49.912574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.912554 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:15:49.922731 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:49.922680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfl9\" (UniqueName: \"kubernetes.io/projected/3036029f-9306-4f30-a02e-b0d5f00e1cd4-kube-api-access-6sfl9\") pod \"network-check-target-5hbp9\" (UID: \"3036029f-9306-4f30-a02e-b0d5f00e1cd4\") " pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:50.034488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.034416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d9crx"] Jul 06 20:15:50.037756 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:50.037730 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a9039_dd79_406d_96c8_0e10737ecf6d.slice/crio-1329fac721f3096805546d48903faf48ac5b01e9559940c1ce6b46d1e9572cb1 WatchSource:0}: Error finding container 1329fac721f3096805546d48903faf48ac5b01e9559940c1ce6b46d1e9572cb1: Status 404 returned error can't find the container with id 1329fac721f3096805546d48903faf48ac5b01e9559940c1ce6b46d1e9572cb1 Jul 06 20:15:50.049206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.049153 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mpq6z"] Jul 06 20:15:50.052022 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:50.052003 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd1d381_faee_4d65_9d12_05c0a91abc96.slice/crio-b17ee2aada0b7dcf779a69b5446162a02578dd9a6ad519267c8940fa762b4780 WatchSource:0}: Error finding container b17ee2aada0b7dcf779a69b5446162a02578dd9a6ad519267c8940fa762b4780: Status 404 returned error can't find the container with id b17ee2aada0b7dcf779a69b5446162a02578dd9a6ad519267c8940fa762b4780 Jul 06 20:15:50.184352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.184320 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8g8p6\"" Jul 06 20:15:50.193194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.193172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:50.310011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.309955 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pxw7m" Jul 06 20:15:50.312122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.312097 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hbp9"] Jul 06 20:15:50.315186 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:50.315165 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3036029f_9306_4f30_a02e_b0d5f00e1cd4.slice/crio-ebda6dd36b9b70956f89fc478db9197009cc968e83d198ba633086495c43f2c3 WatchSource:0}: Error finding container ebda6dd36b9b70956f89fc478db9197009cc968e83d198ba633086495c43f2c3: Status 404 returned error can't find the container with id ebda6dd36b9b70956f89fc478db9197009cc968e83d198ba633086495c43f2c3 Jul 06 20:15:50.341426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.341389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9crx" event={"ID":"de5a9039-dd79-406d-96c8-0e10737ecf6d","Type":"ContainerStarted","Data":"1329fac721f3096805546d48903faf48ac5b01e9559940c1ce6b46d1e9572cb1"} Jul 06 20:15:50.342691 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.342664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mpq6z" event={"ID":"5fd1d381-faee-4d65-9d12-05c0a91abc96","Type":"ContainerStarted","Data":"b17ee2aada0b7dcf779a69b5446162a02578dd9a6ad519267c8940fa762b4780"} Jul 06 20:15:50.344401 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.344379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glwkf" event={"ID":"c8be60e6-c948-4081-a1ea-e1fe0012e849","Type":"ContainerStarted","Data":"9b4ed3c003afbc910d6020f8fc393f8c96890a373d6a08bab339b48240a85799"} Jul 06 20:15:50.345506 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.345485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hbp9" event={"ID":"3036029f-9306-4f30-a02e-b0d5f00e1cd4","Type":"ContainerStarted","Data":"ebda6dd36b9b70956f89fc478db9197009cc968e83d198ba633086495c43f2c3"} Jul 06 20:15:50.502174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:50.502142 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbhmr_e0d27bad-d36b-409c-afdd-8bc6ac3a8e91/serve-healthcheck-canary/0.log" Jul 06 20:15:51.354270 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:51.354206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glwkf" event={"ID":"c8be60e6-c948-4081-a1ea-e1fe0012e849","Type":"ContainerStarted","Data":"47e20087aee42242a430cd896ccd4345b9571aef16eb6d99991841687fca5fdf"} Jul 06 20:15:51.884318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:51.884235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:52.196878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.196843 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:52.295376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.295338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:52.365562 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.365523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9crx" event={"ID":"de5a9039-dd79-406d-96c8-0e10737ecf6d","Type":"ContainerStarted","Data":"cd8e189cf434a81b2f2ed6f42c2c048367329fd5205810d4a1fb97e542101cae"} Jul 06 20:15:52.366012 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.365620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d9crx" event={"ID":"de5a9039-dd79-406d-96c8-0e10737ecf6d","Type":"ContainerStarted","Data":"8f0cdb4d19fcc6a59c9ff78b3b1f052540f4cf68af8058044eb2eabf306d6844"} Jul 06 20:15:52.384788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.384616 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d9crx" podStartSLOduration=67.05238187 podStartE2EDuration="1m8.384595745s" totalImagesPullingTime="1.332213875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:52.382660088 +0000 UTC m=+68.873283643" watchObservedRunningTime="2026-07-06 20:15:52.384595745 +0000 UTC m=+68.875219296" Jul 06 20:15:52.498119 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.498042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:52.587514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:52.587483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.200939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.200889 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.326413 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.326382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.432880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.432846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.588393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.588314 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2"] Jul 06 20:15:54.590697 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.590668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.592742 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.592713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.595731 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.595711 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:15:54.596296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.596121 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:15:54.596296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.596123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-n8766\"" Jul 06 20:15:54.596663 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.596604 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 20:15:54.596854 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.596840 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 20:15:54.596939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.596849 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:15:54.609568 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.609547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2"] Jul 06 20:15:54.610853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.610831 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr"] Jul 06 20:15:54.613383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.613367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.616143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.616126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 20:15:54.616264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.616215 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8qnkv\"" Jul 06 20:15:54.616264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.616245 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 20:15:54.618488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.618471 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 20:15:54.629678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.629650 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr"] Jul 06 20:15:54.630827 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.630801 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bqg8n"] Jul 06 20:15:54.633547 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.633531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.633727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.633705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75782c50-01a4-43b8-b4af-8afff91c5ead-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.633816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.633755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.633878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.633800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmp9\" (UniqueName: \"kubernetes.io/projected/75782c50-01a4-43b8-b4af-8afff91c5ead-kube-api-access-wrmp9\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.633878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.633870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.637886 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.637863 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:15:54.638094 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.637867 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:15:54.639056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.639038 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:15:54.639388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.639366 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97kgv\"" Jul 06 20:15:54.692017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.691993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:54.735284 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.735403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-metrics-client-ca\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-textfile\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.735403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-sys\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.735408 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.735472 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls podName:75782c50-01a4-43b8-b4af-8afff91c5ead nodeName:}" failed. No retries permitted until 2026-07-06 20:15:55.235451748 +0000 UTC m=+71.726075276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-z22m2" (UID: "75782c50-01a4-43b8-b4af-8afff91c5ead") : secret "openshift-state-metrics-tls" not found Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsfk\" (UniqueName: \"kubernetes.io/projected/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-api-access-tjsfk\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-wtmp\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75782c50-01a4-43b8-b4af-8afff91c5ead-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-root\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.735899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmp9\" (UniqueName: \"kubernetes.io/projected/75782c50-01a4-43b8-b4af-8afff91c5ead-kube-api-access-wrmp9\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.736144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr6b\" (UniqueName: \"kubernetes.io/projected/2387edeb-79d8-49fd-b9fd-b6acab436a83-kube-api-access-hbr6b\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.736144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.736144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.735995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.736326 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.736304 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75782c50-01a4-43b8-b4af-8afff91c5ead-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.738229 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.738208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.753669 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.753645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmp9\" (UniqueName: \"kubernetes.io/projected/75782c50-01a4-43b8-b4af-8afff91c5ead-kube-api-access-wrmp9\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:54.836813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr6b\" (UniqueName: \"kubernetes.io/projected/2387edeb-79d8-49fd-b9fd-b6acab436a83-kube-api-access-hbr6b\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-metrics-client-ca\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-textfile\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.836957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.836946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-sys\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837006 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-sys\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsfk\" (UniqueName: \"kubernetes.io/projected/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-api-access-tjsfk\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-wtmp\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.837251 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 20:15:54.837261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.837329 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls podName:2387edeb-79d8-49fd-b9fd-b6acab436a83 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:55.337308452 +0000 UTC m=+71.827931990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls") pod "node-exporter-bqg8n" (UID: "2387edeb-79d8-49fd-b9fd-b6acab436a83") : secret "node-exporter-tls" not found Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837337 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-textfile\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-root\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-root\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837479 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-wtmp\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837527 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-metrics-client-ca\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.837563 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837591 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837613 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.837641 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:15:54.837621 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls podName:400fd57c-a0e7-4bf1-a060-0ba19f3103b1 nodeName:}" failed. No retries permitted until 2026-07-06 20:15:55.337605798 +0000 UTC m=+71.828229342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-flbkr" (UID: "400fd57c-a0e7-4bf1-a060-0ba19f3103b1") : secret "kube-state-metrics-tls" not found Jul 06 20:15:54.838044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.838044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.837980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.839381 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.839337 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.839574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.839558 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.846784 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.846766 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsfk\" (UniqueName: \"kubernetes.io/projected/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-api-access-tjsfk\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:54.847029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.847013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr6b\" (UniqueName: \"kubernetes.io/projected/2387edeb-79d8-49fd-b9fd-b6acab436a83-kube-api-access-hbr6b\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:54.928092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:54.928062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.029263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.029234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.123898 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.123811 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.146090 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.146064 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:55.150281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.150264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:15:55.195466 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.195433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.240934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.240896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:55.243682 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.243660 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75782c50-01a4-43b8-b4af-8afff91c5ead-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-z22m2\" (UID: \"75782c50-01a4-43b8-b4af-8afff91c5ead\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:55.319362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.319323 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.341593 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.341573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:55.341705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.341612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:55.344080 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.344057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2387edeb-79d8-49fd-b9fd-b6acab436a83-node-exporter-tls\") pod \"node-exporter-bqg8n\" (UID: \"2387edeb-79d8-49fd-b9fd-b6acab436a83\") " pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:55.344172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.344159 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400fd57c-a0e7-4bf1-a060-0ba19f3103b1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-flbkr\" (UID: \"400fd57c-a0e7-4bf1-a060-0ba19f3103b1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:55.416811 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.416736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.502843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.502815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" Jul 06 20:15:55.524273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.524251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" Jul 06 20:15:55.538627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.538601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.546844 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.546825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqg8n" Jul 06 20:15:55.628995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.628962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.748275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.748198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.848564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.848534 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:55.920388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:55.920359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:56.258404 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:56.258364 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2387edeb_79d8_49fd_b9fd_b6acab436a83.slice/crio-63480f66c78699b50a52076ff4df23795df36171a41a5359305bac099b2f8592 WatchSource:0}: Error finding container 63480f66c78699b50a52076ff4df23795df36171a41a5359305bac099b2f8592: Status 404 returned error can't find the container with id 63480f66c78699b50a52076ff4df23795df36171a41a5359305bac099b2f8592 Jul 06 20:15:56.383892 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:56.383642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqg8n" event={"ID":"2387edeb-79d8-49fd-b9fd-b6acab436a83","Type":"ContainerStarted","Data":"63480f66c78699b50a52076ff4df23795df36171a41a5359305bac099b2f8592"} Jul 06 20:15:56.426347 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:56.426256 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr"] Jul 06 20:15:56.428410 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:56.428386 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400fd57c_a0e7_4bf1_a060_0ba19f3103b1.slice/crio-1fe2b75cf3bc7281784ad17f309920b08cba65715c50bc73164581813899aea4 WatchSource:0}: Error finding container 1fe2b75cf3bc7281784ad17f309920b08cba65715c50bc73164581813899aea4: Status 404 returned error can't find the container with id 1fe2b75cf3bc7281784ad17f309920b08cba65715c50bc73164581813899aea4 Jul 06 20:15:56.439810 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:56.439791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2"] Jul 06 20:15:56.442718 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:56.442693 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75782c50_01a4_43b8_b4af_8afff91c5ead.slice/crio-32315093b95e13b7d052173400c4e631d6654128a0f3717f505f0d975402ff41 WatchSource:0}: Error finding container 32315093b95e13b7d052173400c4e631d6654128a0f3717f505f0d975402ff41: Status 404 returned error can't find the container with id 32315093b95e13b7d052173400c4e631d6654128a0f3717f505f0d975402ff41 Jul 06 20:15:57.296314 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.296290 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c9b96759f-dmn9j" Jul 06 20:15:57.387848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.387816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" event={"ID":"400fd57c-a0e7-4bf1-a060-0ba19f3103b1","Type":"ContainerStarted","Data":"1fe2b75cf3bc7281784ad17f309920b08cba65715c50bc73164581813899aea4"} Jul 06 20:15:57.389148 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.389123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mpq6z" event={"ID":"5fd1d381-faee-4d65-9d12-05c0a91abc96","Type":"ContainerStarted","Data":"dcc48cad9bafbbeda5f3a9d81f19649250bd0bc44ad9893b9ef616e802ca5db5"} Jul 06 20:15:57.391083 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.391060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glwkf" event={"ID":"c8be60e6-c948-4081-a1ea-e1fe0012e849","Type":"ContainerStarted","Data":"a9dcd5a1b334c3833b0476884c59870debfdb1dad527784eda891b57c059e0b8"} Jul 06 20:15:57.392772 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.392751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" event={"ID":"75782c50-01a4-43b8-b4af-8afff91c5ead","Type":"ContainerStarted","Data":"2a1ae7f3c49756822e54657f8cb0bc623f72c261863b9778b3f18d84d1a8eace"} Jul 06 20:15:57.392856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.392789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" event={"ID":"75782c50-01a4-43b8-b4af-8afff91c5ead","Type":"ContainerStarted","Data":"8c4a725a63ed602794e7d9abfba02b5280ed4dd9956a464e3d2815dfd8c6fd94"} Jul 06 20:15:57.392856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.392809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" event={"ID":"75782c50-01a4-43b8-b4af-8afff91c5ead","Type":"ContainerStarted","Data":"32315093b95e13b7d052173400c4e631d6654128a0f3717f505f0d975402ff41"} Jul 06 20:15:57.393978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.393961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hbp9" event={"ID":"3036029f-9306-4f30-a02e-b0d5f00e1cd4","Type":"ContainerStarted","Data":"d2da2b91cb149956aa5d45b8bdc19b5ea18d02e4700d3f3b962aaf4db5330d27"} Jul 06 20:15:57.394606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.394584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:15:57.403942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.403873 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mpq6z" podStartSLOduration=67.202239478 podStartE2EDuration="1m13.403860431s" totalImagesPullingTime="6.201620953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:57.402757363 +0000 UTC m=+73.893380950" watchObservedRunningTime="2026-07-06 20:15:57.403860431 +0000 UTC m=+73.894483981" Jul 06 20:15:57.422874 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.422831 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5hbp9" podStartSLOduration=67.482743679 podStartE2EDuration="1m13.422814307s" totalImagesPullingTime="5.940070628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:57.421618103 +0000 UTC m=+73.912241654" watchObservedRunningTime="2026-07-06 20:15:57.422814307 +0000 UTC m=+73.913437850" Jul 06 20:15:57.423356 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.423329 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:57.441918 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.441861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-glwkf" podStartSLOduration=3.551998597 podStartE2EDuration="13.441844066s" totalImagesPullingTime="9.889845469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:15:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:57.440262279 +0000 UTC m=+73.930885830" watchObservedRunningTime="2026-07-06 20:15:57.441844066 +0000 UTC m=+73.932467617" Jul 06 20:15:57.517188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.517160 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:57.575499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.575431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:57.937016 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:57.936985 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:58.216980 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.216888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:58.401224 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.401191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqg8n" event={"ID":"2387edeb-79d8-49fd-b9fd-b6acab436a83","Type":"ContainerStarted","Data":"cad2ea94f5a7cb0aa82894ef4cc715aedd36924d48c51e5346af65fdf119059a"} Jul 06 20:15:58.404343 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.404310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" event={"ID":"75782c50-01a4-43b8-b4af-8afff91c5ead","Type":"ContainerStarted","Data":"2854b2ab16b2cc3bbcd08fa455c11b5f501d49dc3617cbea7d0323bf508e56cb"} Jul 06 20:15:58.407092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.406813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" event={"ID":"400fd57c-a0e7-4bf1-a060-0ba19f3103b1","Type":"ContainerStarted","Data":"1b31c967d8557f9312cc5567179be0030e968faa9238bee7544669e10141d966"} Jul 06 20:15:58.722855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.722783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:58.726505 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.726483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56776464-p45nk" Jul 06 20:15:58.741635 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.741588 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-z22m2" podStartSLOduration=3.067476633 podStartE2EDuration="4.741574438s" totalImagesPullingTime="1.674097805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:58.45349715 +0000 UTC m=+74.944120701" watchObservedRunningTime="2026-07-06 20:15:58.741574438 +0000 UTC m=+75.232197988" Jul 06 20:15:58.789519 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.789490 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:15:58.903308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.903277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:58.973647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.973577 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:58.980130 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.980105 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f5db85cbc-4gzqt"] Jul 06 20:15:58.983163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.983149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:58.985498 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.985472 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-873hlsgianvhn\"" Jul 06 20:15:58.985873 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.985842 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 20:15:58.986071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.986049 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6h47p\"" Jul 06 20:15:58.986259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.986238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 20:15:58.986388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.986364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 20:15:58.986929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.986582 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 20:15:58.994176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:58.994153 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f5db85cbc-4gzqt"] Jul 06 20:15:59.073157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-metrics-server-audit-profiles\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-client-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20896075-806c-45f8-8c5e-2454448b8adc-audit-log\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-client-certs\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkfd\" (UniqueName: \"kubernetes.io/projected/20896075-806c-45f8-8c5e-2454448b8adc-kube-api-access-tkkfd\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.073492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.073363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-tls\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.106998 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.106965 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:59.163674 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.163649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:59.173732 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-tls\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.173791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-metrics-server-audit-profiles\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.173825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.173825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-client-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.173893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20896075-806c-45f8-8c5e-2454448b8adc-audit-log\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.174004 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.173982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-client-certs\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.174076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.174029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkfd\" (UniqueName: \"kubernetes.io/projected/20896075-806c-45f8-8c5e-2454448b8adc-kube-api-access-tkkfd\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.174289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.174266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20896075-806c-45f8-8c5e-2454448b8adc-audit-log\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.174734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.174709 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.174734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.174720 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20896075-806c-45f8-8c5e-2454448b8adc-metrics-server-audit-profiles\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.176138 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.176112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-client-ca-bundle\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.176627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.176604 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-client-certs\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.176682 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.176644 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20896075-806c-45f8-8c5e-2454448b8adc-secret-metrics-server-tls\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.182033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.182016 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkfd\" (UniqueName: \"kubernetes.io/projected/20896075-806c-45f8-8c5e-2454448b8adc-kube-api-access-tkkfd\") pod \"metrics-server-f5db85cbc-4gzqt\" (UID: \"20896075-806c-45f8-8c5e-2454448b8adc\") " pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.296764 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.296700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:15:59.381587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.381557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:59.411689 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.411660 2574 generic.go:356] "Generic (PLEG): container finished" podID="2387edeb-79d8-49fd-b9fd-b6acab436a83" containerID="cad2ea94f5a7cb0aa82894ef4cc715aedd36924d48c51e5346af65fdf119059a" exitCode=0 Jul 06 20:15:59.412061 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.411744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqg8n" event={"ID":"2387edeb-79d8-49fd-b9fd-b6acab436a83","Type":"ContainerDied","Data":"cad2ea94f5a7cb0aa82894ef4cc715aedd36924d48c51e5346af65fdf119059a"} Jul 06 20:15:59.414005 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.413968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" event={"ID":"400fd57c-a0e7-4bf1-a060-0ba19f3103b1","Type":"ContainerStarted","Data":"8690bfda449c3eb667eb23dd7d204e48eaf107516967c55703d61d0546197217"} Jul 06 20:15:59.414118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.414037 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" event={"ID":"400fd57c-a0e7-4bf1-a060-0ba19f3103b1","Type":"ContainerStarted","Data":"b0f2e1b8c7c8a3db8b1a31e22a5e753d23227d71d00a09124db26b6bb85c1547"} Jul 06 20:15:59.420001 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.419980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f5db85cbc-4gzqt"] Jul 06 20:15:59.422973 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:15:59.422954 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20896075_806c_45f8_8c5e_2454448b8adc.slice/crio-fdcc46bc570419891f4c8d0f5aef235f60ce8b432e8e6378e7f3862e6c99553f WatchSource:0}: Error finding container fdcc46bc570419891f4c8d0f5aef235f60ce8b432e8e6378e7f3862e6c99553f: Status 404 returned error can't find the container with id fdcc46bc570419891f4c8d0f5aef235f60ce8b432e8e6378e7f3862e6c99553f Jul 06 20:15:59.450920 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.450858 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-flbkr" podStartSLOduration=3.582083433 podStartE2EDuration="5.450839022s" totalImagesPullingTime="1.868755589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:59.450551118 +0000 UTC m=+75.941174671" watchObservedRunningTime="2026-07-06 20:15:59.450839022 +0000 UTC m=+75.941462575" Jul 06 20:15:59.457614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.457591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:59.837063 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.837032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:15:59.969247 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:15:59.969215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:00.069300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.069266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:00.420185 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.420146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqg8n" event={"ID":"2387edeb-79d8-49fd-b9fd-b6acab436a83","Type":"ContainerStarted","Data":"20a20efd786b82a28eb33de10ba59d33aeb99989d66bab32b235991ec4878624"} Jul 06 20:16:00.420633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.420225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqg8n" event={"ID":"2387edeb-79d8-49fd-b9fd-b6acab436a83","Type":"ContainerStarted","Data":"4befc8a14167d653bfeb9148a8f78cc6671f3ee2f53dbcb0c0686736ec2e4e02"} Jul 06 20:16:00.421454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.421431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" event={"ID":"20896075-806c-45f8-8c5e-2454448b8adc","Type":"ContainerStarted","Data":"fdcc46bc570419891f4c8d0f5aef235f60ce8b432e8e6378e7f3862e6c99553f"} Jul 06 20:16:00.440642 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.440573 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bqg8n" podStartSLOduration=4.3128548460000005 podStartE2EDuration="6.440557962s" totalImagesPullingTime="2.039056985s" totalInitContainerRuntime="88.646131ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:16:00.438965369 +0000 UTC m=+76.929588923" watchObservedRunningTime="2026-07-06 20:16:00.440557962 +0000 UTC m=+76.931181511" Jul 06 20:16:00.804840 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:00.804745 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:01.047084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.047050 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:01.050348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.050327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.059327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.059303 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:01.191756 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.191872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdzr\" (UniqueName: \"kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.191872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.191872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.192006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.192006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.191979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.192068 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.192007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdzr\" (UniqueName: \"kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.292473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.292469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.293214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.293194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.293319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.293299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.293365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.293315 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.293365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.293321 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.294827 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.294803 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.295033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.295017 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.300673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.300653 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdzr\" (UniqueName: \"kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr\") pod \"console-869b97698c-kzx5n\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.360204 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.360161 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:01.435327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.435292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" event={"ID":"20896075-806c-45f8-8c5e-2454448b8adc","Type":"ContainerStarted","Data":"b2808eed44e223a3ffeb16b3c20aa3e7438678f2bd0a3b96e47b6ece412eeb5e"} Jul 06 20:16:01.459170 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.459113 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" podStartSLOduration=1.808164284 podStartE2EDuration="3.45909104s" totalImagesPullingTime="1.650926756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:16:01.457481482 +0000 UTC m=+77.948105032" watchObservedRunningTime="2026-07-06 20:16:01.45909104 +0000 UTC m=+77.949714591" Jul 06 20:16:01.487454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.487421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:01.510880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.510853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:01.513847 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:16:01.513823 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7793100b_1b79_457a_91c4_1408a73a1565.slice/crio-4728d95f108bfcd55069a37c5c89196d34ae4d7236a482c95dab8f04e61aeb8b WatchSource:0}: Error finding container 4728d95f108bfcd55069a37c5c89196d34ae4d7236a482c95dab8f04e61aeb8b: Status 404 returned error can't find the container with id 4728d95f108bfcd55069a37c5c89196d34ae4d7236a482c95dab8f04e61aeb8b Jul 06 20:16:01.590256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:01.590230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:02.439856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:02.439824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869b97698c-kzx5n" event={"ID":"7793100b-1b79-457a-91c4-1408a73a1565","Type":"ContainerStarted","Data":"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc"} Jul 06 20:16:02.440296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:02.439873 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869b97698c-kzx5n" event={"ID":"7793100b-1b79-457a-91c4-1408a73a1565","Type":"ContainerStarted","Data":"4728d95f108bfcd55069a37c5c89196d34ae4d7236a482c95dab8f04e61aeb8b"} Jul 06 20:16:03.272352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:03.272305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:03.384943 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:03.384892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.214785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.214646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.281137 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.281082 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-869b97698c-kzx5n" podStartSLOduration=5.281065628 podStartE2EDuration="5.281065628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:16:02.467960124 +0000 UTC m=+78.958583674" watchObservedRunningTime="2026-07-06 20:16:06.281065628 +0000 UTC m=+82.771689177" Jul 06 20:16:06.281584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.281571 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:06.295345 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.295320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.324600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.324579 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:16:06.328583 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.328566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.341044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.341022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:16:06.429918 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.429881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.429923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.429942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9652h\" (UniqueName: \"kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.430020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.430067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.430102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.430276 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.430178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.479349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.479300 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.530728 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.530796 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.530796 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.530886 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9652h\" (UniqueName: \"kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.530962 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.530970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.531004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.531458 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.531478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531629 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.531567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.531744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.531724 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.533094 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.533076 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.533225 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.533211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.537876 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.537860 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9652h\" (UniqueName: \"kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h\") pod \"console-5f95dd7c5c-znrjp\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.580731 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.580706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:06.637822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.637798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:06.761299 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:06.761130 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:16:06.763981 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:16:06.763955 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9520cc_ccb3_41c6_8ae3_6e2d25beec74.slice/crio-2feb4758ff5e8f487b7819e2bd9397386bd3db0682f3c117f7497808a79010d1 WatchSource:0}: Error finding container 2feb4758ff5e8f487b7819e2bd9397386bd3db0682f3c117f7497808a79010d1: Status 404 returned error can't find the container with id 2feb4758ff5e8f487b7819e2bd9397386bd3db0682f3c117f7497808a79010d1 Jul 06 20:16:07.077838 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:07.077744 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:07.132811 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:07.132765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:07.460006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:07.459977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95dd7c5c-znrjp" event={"ID":"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74","Type":"ContainerStarted","Data":"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9"} Jul 06 20:16:07.460355 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:07.460024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95dd7c5c-znrjp" event={"ID":"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74","Type":"ContainerStarted","Data":"2feb4758ff5e8f487b7819e2bd9397386bd3db0682f3c117f7497808a79010d1"} Jul 06 20:16:07.481705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:07.481662 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f95dd7c5c-znrjp" podStartSLOduration=1.481648378 podStartE2EDuration="1.481648378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:16:07.480184254 +0000 UTC m=+83.970807806" watchObservedRunningTime="2026-07-06 20:16:07.481648378 +0000 UTC m=+83.972271960" Jul 06 20:16:08.895946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:08.895878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:08.996254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:08.996225 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:11.360662 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:11.360625 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:15.520057 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:15.520020 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:15.594029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:15.594005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:16.638767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:16.638730 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:16.639251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:16.638786 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:16.643332 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:16.643311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:17.496803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:17.496777 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:16:17.551600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:17.551572 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:16:17.677964 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:17.677939 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:17.780428 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:17.780358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:17.840269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:17.840246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:19.297650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:19.297621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:16:19.298117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:19.297680 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:16:23.743085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:23.743051 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:23.812385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:23.812351 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-559d66d65b-5g2g6" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerName="console" containerID="cri-o://bf9c59ef9a4a1aca3522531df437e900c4dea176723939979452858677978ef9" gracePeriod=15 Jul 06 20:16:23.853595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:23.853564 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:23.919799 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:23.919780 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:24.519349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.519321 2574 generic.go:356] "Generic (PLEG): container finished" podID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerID="bf9c59ef9a4a1aca3522531df437e900c4dea176723939979452858677978ef9" exitCode=0 Jul 06 20:16:24.519470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.519397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559d66d65b-5g2g6" event={"ID":"a534f7b6-75ce-40a3-ad2d-fdedc2044eba","Type":"ContainerDied","Data":"bf9c59ef9a4a1aca3522531df437e900c4dea176723939979452858677978ef9"} Jul 06 20:16:24.573239 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.573215 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:16:24.650610 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650589 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.650740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650636 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.650740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650690 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.650848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650740 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.650848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650770 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cts6n\" (UniqueName: \"kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.650848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.650800 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca\") pod \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\" (UID: \"a534f7b6-75ce-40a3-ad2d-fdedc2044eba\") " Jul 06 20:16:24.651097 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.651072 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config" (OuterVolumeSpecName: "console-config") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:24.651146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.651125 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:24.651232 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.651209 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca" (OuterVolumeSpecName: "service-ca") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:24.652813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.652789 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:24.652937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.652860 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:24.652937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.652887 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n" (OuterVolumeSpecName: "kube-api-access-cts6n") pod "a534f7b6-75ce-40a3-ad2d-fdedc2044eba" (UID: "a534f7b6-75ce-40a3-ad2d-fdedc2044eba"). InnerVolumeSpecName "kube-api-access-cts6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:16:24.752269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752222 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:24.752269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752242 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:24.752269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752251 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cts6n\" (UniqueName: \"kubernetes.io/projected/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-kube-api-access-cts6n\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:24.752269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752261 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:24.752269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752269 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:24.752639 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:24.752277 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a534f7b6-75ce-40a3-ad2d-fdedc2044eba-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:25.524064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:25.524035 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559d66d65b-5g2g6" Jul 06 20:16:25.524064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:25.524042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559d66d65b-5g2g6" event={"ID":"a534f7b6-75ce-40a3-ad2d-fdedc2044eba","Type":"ContainerDied","Data":"120a556b26f51f6860fe62a6c80192d7b53c2052324f71ebe41b64d9e8c53223"} Jul 06 20:16:25.524261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:25.524092 2574 scope.go:117] "RemoveContainer" containerID="bf9c59ef9a4a1aca3522531df437e900c4dea176723939979452858677978ef9" Jul 06 20:16:25.543871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:25.543853 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:16:25.547596 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:25.547576 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559d66d65b-5g2g6"] Jul 06 20:16:26.071024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:26.070994 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" path="/var/lib/kubelet/pods/a534f7b6-75ce-40a3-ad2d-fdedc2044eba/volumes" Jul 06 20:16:28.409800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:28.409772 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5hbp9" Jul 06 20:16:30.501432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:30.501400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:31.302007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.301956 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-869b97698c-kzx5n" podUID="7793100b-1b79-457a-91c4-1408a73a1565" containerName="console" containerID="cri-o://5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc" gracePeriod=15 Jul 06 20:16:31.532188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.532166 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:31.546890 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.546862 2574 generic.go:356] "Generic (PLEG): container finished" podID="7793100b-1b79-457a-91c4-1408a73a1565" containerID="5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc" exitCode=0 Jul 06 20:16:31.547007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.546935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869b97698c-kzx5n" event={"ID":"7793100b-1b79-457a-91c4-1408a73a1565","Type":"ContainerDied","Data":"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc"} Jul 06 20:16:31.547007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.546988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-869b97698c-kzx5n" event={"ID":"7793100b-1b79-457a-91c4-1408a73a1565","Type":"ContainerDied","Data":"4728d95f108bfcd55069a37c5c89196d34ae4d7236a482c95dab8f04e61aeb8b"} Jul 06 20:16:31.547080 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.547009 2574 scope.go:117] "RemoveContainer" containerID="5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc" Jul 06 20:16:31.547080 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.546946 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-869b97698c-kzx5n" Jul 06 20:16:31.555362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.555346 2574 scope.go:117] "RemoveContainer" containerID="5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc" Jul 06 20:16:31.555647 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:16:31.555619 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc\": container with ID starting with 5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc not found: ID does not exist" containerID="5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc" Jul 06 20:16:31.555720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.555652 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc"} err="failed to get container status \"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc\": rpc error: code = NotFound desc = could not find container \"5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc\": container with ID starting with 5d6b7b02cc97f86a7f0b7e0765edb13c6e85f7c46729b072ac37bec33c76a8dc not found: ID does not exist" Jul 06 20:16:31.700188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700170 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700228 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700249 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700383 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700441 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tdzr\" (UniqueName: \"kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr\") pod \"7793100b-1b79-457a-91c4-1408a73a1565\" (UID: \"7793100b-1b79-457a-91c4-1408a73a1565\") " Jul 06 20:16:31.700612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700577 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca" (OuterVolumeSpecName: "service-ca") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:31.700669 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700616 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:31.700715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700673 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.700715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700692 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.700787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700636 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:31.700821 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.700795 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config" (OuterVolumeSpecName: "console-config") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:31.702364 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.702335 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:31.702442 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.702354 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:31.702442 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.702401 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr" (OuterVolumeSpecName: "kube-api-access-5tdzr") pod "7793100b-1b79-457a-91c4-1408a73a1565" (UID: "7793100b-1b79-457a-91c4-1408a73a1565"). InnerVolumeSpecName "kube-api-access-5tdzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:16:31.801394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.801364 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.801394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.801391 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.801394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.801400 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7793100b-1b79-457a-91c4-1408a73a1565-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.801542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.801408 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7793100b-1b79-457a-91c4-1408a73a1565-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.801542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.801417 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdzr\" (UniqueName: \"kubernetes.io/projected/7793100b-1b79-457a-91c4-1408a73a1565-kube-api-access-5tdzr\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:31.866212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.866189 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:31.872361 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.872342 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-869b97698c-kzx5n"] Jul 06 20:16:31.916580 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:31.916557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:32.074794 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:32.074756 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7793100b-1b79-457a-91c4-1408a73a1565" path="/var/lib/kubelet/pods/7793100b-1b79-457a-91c4-1408a73a1565/volumes" Jul 06 20:16:37.696437 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:37.696402 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:37.775685 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:37.775650 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:16:39.301783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:39.301755 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:16:39.305348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:39.305326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f5db85cbc-4gzqt" Jul 06 20:16:42.571147 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.571110 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-56776464-p45nk" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerName="console" containerID="cri-o://9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155" gracePeriod=15 Jul 06 20:16:42.799291 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.799268 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56776464-p45nk" Jul 06 20:16:42.976440 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976409 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976450 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976511 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976563 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h765n\" (UniqueName: \"kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976603 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert\") pod \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\" (UID: \"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6\") " Jul 06 20:16:42.976937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976857 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca" (OuterVolumeSpecName: "service-ca") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:42.977017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.976973 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config" (OuterVolumeSpecName: "console-config") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:42.977102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.977079 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:42.977198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.977171 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:16:42.978651 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.978631 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:42.978745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.978708 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:16:42.978784 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:42.978747 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n" (OuterVolumeSpecName: "kube-api-access-h765n") pod "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" (UID: "457f825f-ce6f-4f32-a754-fc1d4ddfa3b6"). InnerVolumeSpecName "kube-api-access-h765n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:16:43.077470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077440 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077470 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077480 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077488 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h765n\" (UniqueName: \"kubernetes.io/projected/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-kube-api-access-h765n\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077497 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077505 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.077587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.077513 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:16:43.587788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.587759 2574 generic.go:356] "Generic (PLEG): container finished" podID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerID="9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155" exitCode=0 Jul 06 20:16:43.588098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.587800 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56776464-p45nk" Jul 06 20:16:43.588098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.587847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56776464-p45nk" event={"ID":"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6","Type":"ContainerDied","Data":"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155"} Jul 06 20:16:43.588098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.587887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56776464-p45nk" event={"ID":"457f825f-ce6f-4f32-a754-fc1d4ddfa3b6","Type":"ContainerDied","Data":"c2c088f0a3a604d8f594ad3a5d29c44eb000d2ecb892f225dfef127e9c5d187e"} Jul 06 20:16:43.588098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.587919 2574 scope.go:117] "RemoveContainer" containerID="9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155" Jul 06 20:16:43.595712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.595688 2574 scope.go:117] "RemoveContainer" containerID="9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155" Jul 06 20:16:43.595960 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:16:43.595943 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155\": container with ID starting with 9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155 not found: ID does not exist" containerID="9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155" Jul 06 20:16:43.596028 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.595974 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155"} err="failed to get container status \"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155\": rpc error: code = NotFound desc = could not find container \"9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155\": container with ID starting with 9a19bfcc3f6d0f44acb24600133f15a1c5a70b25cf3d99d795ff1e388bcc6155 not found: ID does not exist" Jul 06 20:16:43.608246 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.608226 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:16:43.613404 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:43.613386 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56776464-p45nk"] Jul 06 20:16:44.072620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:44.072591 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" path="/var/lib/kubelet/pods/457f825f-ce6f-4f32-a754-fc1d4ddfa3b6/volumes" Jul 06 20:16:48.826118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:16:48.826085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:10.694748 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:10.694721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:10.750824 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:10.750801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:14.548482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:14.548448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:14.607275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:14.607248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:16.813395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813365 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813615 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813626 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813646 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7793100b-1b79-457a-91c4-1408a73a1565" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813652 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7793100b-1b79-457a-91c4-1408a73a1565" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813658 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813663 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813701 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="457f825f-ce6f-4f32-a754-fc1d4ddfa3b6" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813709 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a534f7b6-75ce-40a3-ad2d-fdedc2044eba" containerName="console" Jul 06 20:17:16.813734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.813715 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7793100b-1b79-457a-91c4-1408a73a1565" containerName="console" Jul 06 20:17:16.815494 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.815476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.830443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.830424 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:17:16.872569 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.872525 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:16.898839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.898947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.898947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxfg\" (UniqueName: \"kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.898947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.899045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.899045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.898978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.899045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.899028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.938245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.938221 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:16.999524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxfg\" (UniqueName: \"kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:16.999849 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:16.999642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.000378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.000357 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.000490 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.000428 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.000555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.000530 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.000673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.000657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.002045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.002024 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.002393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.002375 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.007369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.007349 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxfg\" (UniqueName: \"kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg\") pod \"console-56f4b5f784-wh7kq\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.016669 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.016649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:17.124723 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.124666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:17.260027 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.259996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:17:17.262803 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:17:17.262775 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980bcd1d_5a1c_4a78_a205_79894d60a24a.slice/crio-c12f07841f3a2730460d51a4852fcf31e51c17b0ea52cd5c7f66850cf7883875 WatchSource:0}: Error finding container c12f07841f3a2730460d51a4852fcf31e51c17b0ea52cd5c7f66850cf7883875: Status 404 returned error can't find the container with id c12f07841f3a2730460d51a4852fcf31e51c17b0ea52cd5c7f66850cf7883875 Jul 06 20:17:17.612473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.612445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:17.703860 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.703835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f4b5f784-wh7kq" event={"ID":"980bcd1d-5a1c-4a78-a205-79894d60a24a","Type":"ContainerStarted","Data":"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e"} Jul 06 20:17:17.703991 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.703885 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f4b5f784-wh7kq" event={"ID":"980bcd1d-5a1c-4a78-a205-79894d60a24a","Type":"ContainerStarted","Data":"c12f07841f3a2730460d51a4852fcf31e51c17b0ea52cd5c7f66850cf7883875"} Jul 06 20:17:17.713105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.713084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:17.723883 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:17.723842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56f4b5f784-wh7kq" podStartSLOduration=1.723829545 podStartE2EDuration="1.723829545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:17:17.722471252 +0000 UTC m=+154.213094806" watchObservedRunningTime="2026-07-06 20:17:17.723829545 +0000 UTC m=+154.214453092" Jul 06 20:17:22.610089 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:22.610058 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:22.662574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:22.662551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:27.125363 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.125324 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:27.125742 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.125416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:27.131025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.131004 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:27.739611 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.739579 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:17:27.785131 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.785105 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:17:27.890529 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.890499 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:27.947978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:27.947948 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:28.014098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:28.014035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:30.507316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:30.507277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:30.608077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:30.608045 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:17:52.804767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:52.804671 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f95dd7c5c-znrjp" podUID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" containerName="console" containerID="cri-o://285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9" gracePeriod=15 Jul 06 20:17:53.043345 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.043320 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:17:53.144887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.144830 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.144887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.144876 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.144940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.144960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9652h\" (UniqueName: \"kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.144986 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145021 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca\") pod \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\" (UID: \"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74\") " Jul 06 20:17:53.145441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145413 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:53.145587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145423 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config" (OuterVolumeSpecName: "console-config") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:53.145587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145425 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:53.145587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.145487 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:53.146968 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.146943 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:17:53.147115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.147094 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:17:53.147159 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.147096 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h" (OuterVolumeSpecName: "kube-api-access-9652h") pod "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" (UID: "9a9520cc-ccb3-41c6-8ae3-6e2d25beec74"). InnerVolumeSpecName "kube-api-access-9652h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:17:53.246507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246486 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246505 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246515 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9652h\" (UniqueName: \"kubernetes.io/projected/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-kube-api-access-9652h\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246524 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246533 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246542 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.246619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.246550 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:17:53.824215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.824191 2574 generic.go:356] "Generic (PLEG): container finished" podID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" containerID="285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9" exitCode=0 Jul 06 20:17:53.824518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.824232 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f95dd7c5c-znrjp" Jul 06 20:17:53.824518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.824261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95dd7c5c-znrjp" event={"ID":"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74","Type":"ContainerDied","Data":"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9"} Jul 06 20:17:53.824518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.824299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f95dd7c5c-znrjp" event={"ID":"9a9520cc-ccb3-41c6-8ae3-6e2d25beec74","Type":"ContainerDied","Data":"2feb4758ff5e8f487b7819e2bd9397386bd3db0682f3c117f7497808a79010d1"} Jul 06 20:17:53.824518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.824317 2574 scope.go:117] "RemoveContainer" containerID="285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9" Jul 06 20:17:53.832021 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.832006 2574 scope.go:117] "RemoveContainer" containerID="285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9" Jul 06 20:17:53.832239 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:17:53.832223 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9\": container with ID starting with 285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9 not found: ID does not exist" containerID="285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9" Jul 06 20:17:53.832285 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.832250 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9"} err="failed to get container status \"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9\": rpc error: code = NotFound desc = could not find container \"285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9\": container with ID starting with 285d7fba8e97e1debf40e373964b99614c7595c55e116ff2b9f87926dd8310a9 not found: ID does not exist" Jul 06 20:17:53.843131 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.843109 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:17:53.847210 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:53.847190 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f95dd7c5c-znrjp"] Jul 06 20:17:54.070566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:54.070542 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" path="/var/lib/kubelet/pods/9a9520cc-ccb3-41c6-8ae3-6e2d25beec74/volumes" Jul 06 20:17:57.571067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:17:57.571032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:14.211155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.211121 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:14.308577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.308546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:14.823411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.823381 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj"] Jul 06 20:18:14.823647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.823636 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" containerName="console" Jul 06 20:18:14.823688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.823648 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" containerName="console" Jul 06 20:18:14.823725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.823689 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9520cc-ccb3-41c6-8ae3-6e2d25beec74" containerName="console" Jul 06 20:18:14.825737 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.825723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.828289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.828260 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:18:14.828412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.828294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:18:14.829182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.829166 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nx8nz\"" Jul 06 20:18:14.834305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.834281 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj"] Jul 06 20:18:14.883810 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.883787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.883942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.883817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.883942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.883840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjv6z\" (UniqueName: \"kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.985204 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.985182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.985303 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.985210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.985303 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.985232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjv6z\" (UniqueName: \"kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.985559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.985541 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.985599 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.985582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:14.994102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:14.994084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjv6z\" (UniqueName: \"kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:15.134411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:15.134337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:15.257197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:15.257166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj"] Jul 06 20:18:15.260052 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:18:15.260026 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce290b2f_02bd_4dfb_9fd0_e2be8277a428.slice/crio-2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5 WatchSource:0}: Error finding container 2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5: Status 404 returned error can't find the container with id 2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5 Jul 06 20:18:15.898844 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:15.898816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerStarted","Data":"2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5"} Jul 06 20:18:24.934672 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:24.934641 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerID="77f6288d1f8450934ab14bb725d6c9dc64d57392e48f898fa51ef3d4abf1f4ff" exitCode=0 Jul 06 20:18:24.935042 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:24.934686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerDied","Data":"77f6288d1f8450934ab14bb725d6c9dc64d57392e48f898fa51ef3d4abf1f4ff"} Jul 06 20:18:26.943636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:26.943609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerStarted","Data":"308b1d4ba70502b84be13d572a521e0e4941801d7dc0e37885096caae261fc7e"} Jul 06 20:18:27.948601 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:27.948567 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerID="308b1d4ba70502b84be13d572a521e0e4941801d7dc0e37885096caae261fc7e" exitCode=0 Jul 06 20:18:27.948965 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:27.948655 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerDied","Data":"308b1d4ba70502b84be13d572a521e0e4941801d7dc0e37885096caae261fc7e"} Jul 06 20:18:30.597432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:30.597398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:35.979212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:35.979174 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerStarted","Data":"5b411b1d0968d9e19a3bf6adff055cd862ff5d29a516841a7e17b31023cecd78"} Jul 06 20:18:35.998806 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:35.998757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" podStartSLOduration=3.162387839 podStartE2EDuration="21.998740725s" totalImagesPullingTime="18.836352886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:18:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:18:35.997625535 +0000 UTC m=+232.488249085" watchObservedRunningTime="2026-07-06 20:18:35.998740725 +0000 UTC m=+232.489364276" Jul 06 20:18:36.984696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:36.984661 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerID="5b411b1d0968d9e19a3bf6adff055cd862ff5d29a516841a7e17b31023cecd78" exitCode=0 Jul 06 20:18:36.985056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:36.984704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerDied","Data":"5b411b1d0968d9e19a3bf6adff055cd862ff5d29a516841a7e17b31023cecd78"} Jul 06 20:18:38.106192 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.106167 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:38.260958 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.260874 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle\") pod \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " Jul 06 20:18:38.260958 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.260936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjv6z\" (UniqueName: \"kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z\") pod \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " Jul 06 20:18:38.261150 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.260964 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util\") pod \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\" (UID: \"ce290b2f-02bd-4dfb-9fd0-e2be8277a428\") " Jul 06 20:18:38.261540 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.261512 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle" (OuterVolumeSpecName: "bundle") pod "ce290b2f-02bd-4dfb-9fd0-e2be8277a428" (UID: "ce290b2f-02bd-4dfb-9fd0-e2be8277a428"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:18:38.263004 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.262982 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z" (OuterVolumeSpecName: "kube-api-access-sjv6z") pod "ce290b2f-02bd-4dfb-9fd0-e2be8277a428" (UID: "ce290b2f-02bd-4dfb-9fd0-e2be8277a428"). InnerVolumeSpecName "kube-api-access-sjv6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:18:38.265145 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.265122 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util" (OuterVolumeSpecName: "util") pod "ce290b2f-02bd-4dfb-9fd0-e2be8277a428" (UID: "ce290b2f-02bd-4dfb-9fd0-e2be8277a428"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:18:38.362188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.362163 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:38.362188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.362185 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjv6z\" (UniqueName: \"kubernetes.io/projected/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-kube-api-access-sjv6z\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:38.362310 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.362195 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce290b2f-02bd-4dfb-9fd0-e2be8277a428-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:38.993362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.993330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" event={"ID":"ce290b2f-02bd-4dfb-9fd0-e2be8277a428","Type":"ContainerDied","Data":"2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5"} Jul 06 20:18:38.993462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.993369 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406ltjj" Jul 06 20:18:38.993462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:38.993401 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2deb03923c643723bf96125d9a32c1a6edd849df28ba1c039c20aece1ea132c5" Jul 06 20:18:40.861120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:40.861085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:41.572493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:41.572464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:41.711470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:41.711439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:42.815143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815114 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465"] Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815348 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="pull" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815359 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="pull" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815368 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="util" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815373 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="util" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815379 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="extract" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815384 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="extract" Jul 06 20:18:42.815487 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.815430 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce290b2f-02bd-4dfb-9fd0-e2be8277a428" containerName="extract" Jul 06 20:18:42.856156 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.856134 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465"] Jul 06 20:18:42.856276 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.856219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:42.858677 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.858656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 20:18:42.858772 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.858660 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:18:42.858772 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.858731 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zdwsb\"" Jul 06 20:18:42.993078 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.993057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2bq\" (UniqueName: \"kubernetes.io/projected/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-kube-api-access-wj2bq\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:42.993183 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:42.993095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.093699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.093642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2bq\" (UniqueName: \"kubernetes.io/projected/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-kube-api-access-wj2bq\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.093699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.093682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.094239 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.094215 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.101917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.101876 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2bq\" (UniqueName: \"kubernetes.io/projected/40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef-kube-api-access-wj2bq\") pod \"cert-manager-operator-controller-manager-55c444bf86-8x465\" (UID: \"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.165644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.165623 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" Jul 06 20:18:43.305831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:43.305799 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465"] Jul 06 20:18:43.308394 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:18:43.308364 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e6dffb_c6a7_4d6f_bbc2_6d20f6f39cef.slice/crio-d3f88d60e019c249216956c88e3e2fe603dc2cadc301dc4aff00a8857bdd7fc9 WatchSource:0}: Error finding container d3f88d60e019c249216956c88e3e2fe603dc2cadc301dc4aff00a8857bdd7fc9: Status 404 returned error can't find the container with id d3f88d60e019c249216956c88e3e2fe603dc2cadc301dc4aff00a8857bdd7fc9 Jul 06 20:18:44.010071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:44.010038 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" event={"ID":"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef","Type":"ContainerStarted","Data":"d3f88d60e019c249216956c88e3e2fe603dc2cadc301dc4aff00a8857bdd7fc9"} Jul 06 20:18:44.701469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:44.701436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:44.799188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:44.799155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:46.019990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.019941 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" event={"ID":"40e6dffb-c6a7-4d6f-bbc2-6d20f6f39cef","Type":"ContainerStarted","Data":"32c859259dc9e83fcec5cd5accc18f4b834a53141722ea2febb1a8d9e775e7e4"} Jul 06 20:18:46.051543 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.051495 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8x465" podStartSLOduration=2.107354389 podStartE2EDuration="4.05147832s" totalImagesPullingTime="1.944123931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:18:46.050233849 +0000 UTC m=+242.540857401" watchObservedRunningTime="2026-07-06 20:18:46.05147832 +0000 UTC m=+242.542101870" Jul 06 20:18:46.124734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.124695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:46.221067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.221037 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:46.519348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.519321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:46.671880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:46.671854 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:47.138864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.138830 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4"] Jul 06 20:18:47.141191 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.141170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.146007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.145983 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:18:47.146558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.146541 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nx8nz\"" Jul 06 20:18:47.146655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.146576 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:18:47.156836 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.156817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4"] Jul 06 20:18:47.221288 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.221261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.221394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.221312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.221438 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.221405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khttx\" (UniqueName: \"kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.322116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.322088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khttx\" (UniqueName: \"kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.322215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.322123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.322356 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.322334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.322495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.322482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.322617 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.322601 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.332470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.332448 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khttx\" (UniqueName: \"kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.449221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.449198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:47.579132 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:47.579106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4"] Jul 06 20:18:47.581424 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:18:47.581397 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d439b_fcac_464b_9412_a6cb85838ea0.slice/crio-db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623 WatchSource:0}: Error finding container db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623: Status 404 returned error can't find the container with id db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623 Jul 06 20:18:48.032161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:48.032136 2574 generic.go:356] "Generic (PLEG): container finished" podID="367d439b-fcac-464b-9412-a6cb85838ea0" containerID="dce40d580cd810e3c2d1b8f8061edb45b63c108f23bf494870a1b168457e997f" exitCode=0 Jul 06 20:18:48.032311 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:48.032175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" event={"ID":"367d439b-fcac-464b-9412-a6cb85838ea0","Type":"ContainerDied","Data":"dce40d580cd810e3c2d1b8f8061edb45b63c108f23bf494870a1b168457e997f"} Jul 06 20:18:48.032311 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:48.032229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" event={"ID":"367d439b-fcac-464b-9412-a6cb85838ea0","Type":"ContainerStarted","Data":"db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623"} Jul 06 20:18:48.138561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:48.138537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:48.501049 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:48.501021 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:49.139801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.139764 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg"] Jul 06 20:18:49.141994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.141974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.144587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.144569 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 20:18:49.144670 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.144590 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 20:18:49.145409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.145396 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-99pdw\"" Jul 06 20:18:49.154040 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.154019 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg"] Jul 06 20:18:49.237178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.237155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.237263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.237195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfjt\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-kube-api-access-gmfjt\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.338151 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.338127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.338243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.338166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfjt\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-kube-api-access-gmfjt\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.348678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.348652 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.348755 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.348730 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfjt\" (UniqueName: \"kubernetes.io/projected/349fb5cf-7afd-4780-af0f-27e87b05d429-kube-api-access-gmfjt\") pod \"cert-manager-cainjector-559cdd8c87-9rqhg\" (UID: \"349fb5cf-7afd-4780-af0f-27e87b05d429\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.458532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.458512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" Jul 06 20:18:49.578188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:49.578163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg"] Jul 06 20:18:49.580948 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:18:49.580925 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349fb5cf_7afd_4780_af0f_27e87b05d429.slice/crio-8b5515c0bb46eb32789965059038cd14136a8f4d0f2689961e8d3fe4565575a4 WatchSource:0}: Error finding container 8b5515c0bb46eb32789965059038cd14136a8f4d0f2689961e8d3fe4565575a4: Status 404 returned error can't find the container with id 8b5515c0bb46eb32789965059038cd14136a8f4d0f2689961e8d3fe4565575a4 Jul 06 20:18:50.040317 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:50.040287 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" event={"ID":"349fb5cf-7afd-4780-af0f-27e87b05d429","Type":"ContainerStarted","Data":"8b5515c0bb46eb32789965059038cd14136a8f4d0f2689961e8d3fe4565575a4"} Jul 06 20:18:51.270615 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:51.270574 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:51.371945 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:51.371917 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:52.865324 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.865283 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-khk8l"] Jul 06 20:18:52.870684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.870661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:52.873166 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.873142 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qfxdk\"" Jul 06 20:18:52.877477 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.877430 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-khk8l"] Jul 06 20:18:52.958249 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.958222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwwc\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-kube-api-access-ftwwc\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:52.958396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:52.958266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:53.058569 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:53.058536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwwc\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-kube-api-access-ftwwc\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:53.058729 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:53.058582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:53.066896 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:53.066869 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:53.067025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:53.066996 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwwc\" (UniqueName: \"kubernetes.io/projected/2259568b-eb62-435b-851e-50d57fc2a027-kube-api-access-ftwwc\") pod \"cert-manager-webhook-6486ffd997-khk8l\" (UID: \"2259568b-eb62-435b-851e-50d57fc2a027\") " pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:53.182213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:53.182188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:54.439189 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:54.439162 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-khk8l"] Jul 06 20:18:54.441961 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:18:54.441918 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2259568b_eb62_435b_851e_50d57fc2a027.slice/crio-f278c38ac4c7412af5b8ea2dd128709b571d42e4b74171562c99f883bef7092a WatchSource:0}: Error finding container f278c38ac4c7412af5b8ea2dd128709b571d42e4b74171562c99f883bef7092a: Status 404 returned error can't find the container with id f278c38ac4c7412af5b8ea2dd128709b571d42e4b74171562c99f883bef7092a Jul 06 20:18:55.062620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.062581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" event={"ID":"2259568b-eb62-435b-851e-50d57fc2a027","Type":"ContainerStarted","Data":"8ae5428b9c9bfdff2b1ecdcea8931a85b745c22cf41221ebca5c224084875a04"} Jul 06 20:18:55.062620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.062626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" event={"ID":"2259568b-eb62-435b-851e-50d57fc2a027","Type":"ContainerStarted","Data":"f278c38ac4c7412af5b8ea2dd128709b571d42e4b74171562c99f883bef7092a"} Jul 06 20:18:55.062852 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.062687 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:18:55.064463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.064441 2574 generic.go:356] "Generic (PLEG): container finished" podID="367d439b-fcac-464b-9412-a6cb85838ea0" containerID="8f4dad759b4366bbdab12ee9538aa50fce2e42d8e423622038d37aa5de50c544" exitCode=0 Jul 06 20:18:55.064582 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.064479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" event={"ID":"367d439b-fcac-464b-9412-a6cb85838ea0","Type":"ContainerDied","Data":"8f4dad759b4366bbdab12ee9538aa50fce2e42d8e423622038d37aa5de50c544"} Jul 06 20:18:55.066028 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.066008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" event={"ID":"349fb5cf-7afd-4780-af0f-27e87b05d429","Type":"ContainerStarted","Data":"b39bb583432a9028aed3388f13093366256e6b55c9a3f96802073df6f21b33b4"} Jul 06 20:18:55.080976 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.080826 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" podStartSLOduration=3.08080999 podStartE2EDuration="3.08080999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:18:55.079153644 +0000 UTC m=+251.569777195" watchObservedRunningTime="2026-07-06 20:18:55.08080999 +0000 UTC m=+251.571433544" Jul 06 20:18:55.099590 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.099556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rqhg" podStartSLOduration=1.366493555 podStartE2EDuration="6.099543611s" totalImagesPullingTime="4.733050056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:18:55.098150943 +0000 UTC m=+251.588774495" watchObservedRunningTime="2026-07-06 20:18:55.099543611 +0000 UTC m=+251.590167162" Jul 06 20:18:55.154073 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.154035 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:55.282163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:55.282133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:56.071714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:56.071682 2574 generic.go:356] "Generic (PLEG): container finished" podID="367d439b-fcac-464b-9412-a6cb85838ea0" containerID="57ea5b7c1ed3c908be3a3edfc95f2a54ae61bf8af767eaf5838b69ebec9d5866" exitCode=0 Jul 06 20:18:56.072146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:56.071761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" event={"ID":"367d439b-fcac-464b-9412-a6cb85838ea0","Type":"ContainerDied","Data":"57ea5b7c1ed3c908be3a3edfc95f2a54ae61bf8af767eaf5838b69ebec9d5866"} Jul 06 20:18:56.895387 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:56.895356 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:18:57.199231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.198932 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:18:57.283940 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.283917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khttx\" (UniqueName: \"kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx\") pod \"367d439b-fcac-464b-9412-a6cb85838ea0\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " Jul 06 20:18:57.284072 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.283965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util\") pod \"367d439b-fcac-464b-9412-a6cb85838ea0\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " Jul 06 20:18:57.284072 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.283983 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle\") pod \"367d439b-fcac-464b-9412-a6cb85838ea0\" (UID: \"367d439b-fcac-464b-9412-a6cb85838ea0\") " Jul 06 20:18:57.284363 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.284333 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle" (OuterVolumeSpecName: "bundle") pod "367d439b-fcac-464b-9412-a6cb85838ea0" (UID: "367d439b-fcac-464b-9412-a6cb85838ea0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:18:57.285951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.285929 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx" (OuterVolumeSpecName: "kube-api-access-khttx") pod "367d439b-fcac-464b-9412-a6cb85838ea0" (UID: "367d439b-fcac-464b-9412-a6cb85838ea0"). InnerVolumeSpecName "kube-api-access-khttx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:18:57.291274 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.291157 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util" (OuterVolumeSpecName: "util") pod "367d439b-fcac-464b-9412-a6cb85838ea0" (UID: "367d439b-fcac-464b-9412-a6cb85838ea0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:18:57.385079 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.385058 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khttx\" (UniqueName: \"kubernetes.io/projected/367d439b-fcac-464b-9412-a6cb85838ea0-kube-api-access-khttx\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:57.385079 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.385077 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:57.385197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:57.385086 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367d439b-fcac-464b-9412-a6cb85838ea0-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:18:58.081422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:58.081376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" event={"ID":"367d439b-fcac-464b-9412-a6cb85838ea0","Type":"ContainerDied","Data":"db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623"} Jul 06 20:18:58.081537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:58.081466 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6ed620c89da80d88ed8b32a5ae7f5a955dd657fb81b50437db17807e539623" Jul 06 20:18:58.081600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:18:58.081570 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frd7s4" Jul 06 20:19:01.074470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:01.074439 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-khk8l" Jul 06 20:19:02.898845 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:02.898809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:03.009571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:03.009542 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:04.809308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:04.809276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:06.716750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716671 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w72c"] Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716933 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="pull" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716944 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="pull" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716961 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="extract" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716966 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="extract" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716973 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="util" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.716980 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="util" Jul 06 20:19:06.717093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.717061 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="367d439b-fcac-464b-9412-a6cb85838ea0" containerName="extract" Jul 06 20:19:06.734947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.734895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.736524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.736499 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w72c"] Jul 06 20:19:06.737874 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.737855 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jwdbf\"" Jul 06 20:19:06.754954 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.754932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-kube-api-access-nqdnq\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.755051 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.754961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.855321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.855303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-kube-api-access-nqdnq\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.855425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.855329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.872327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.872308 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:06.873254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:06.873229 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/365651bd-7be6-4e35-bfc5-9883a068cb22-kube-api-access-nqdnq\") pod \"cert-manager-55b4d954d5-7w72c\" (UID: \"365651bd-7be6-4e35-bfc5-9883a068cb22\") " pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:07.044542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:07.044495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7w72c" Jul 06 20:19:07.191800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:07.191733 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w72c"] Jul 06 20:19:07.194209 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:07.194183 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod365651bd_7be6_4e35_bfc5_9883a068cb22.slice/crio-60e857ffdfd3a19eebaa352d3409a7ecb691a63d5a0c470ff32cf3bb7c42f63d WatchSource:0}: Error finding container 60e857ffdfd3a19eebaa352d3409a7ecb691a63d5a0c470ff32cf3bb7c42f63d: Status 404 returned error can't find the container with id 60e857ffdfd3a19eebaa352d3409a7ecb691a63d5a0c470ff32cf3bb7c42f63d Jul 06 20:19:08.119727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:08.119695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7w72c" event={"ID":"365651bd-7be6-4e35-bfc5-9883a068cb22","Type":"ContainerStarted","Data":"01beff3ca40afe686662290b396bba3a9fe33d3202dd2065eef1f787da2f85ef"} Jul 06 20:19:08.119727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:08.119732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7w72c" event={"ID":"365651bd-7be6-4e35-bfc5-9883a068cb22","Type":"ContainerStarted","Data":"60e857ffdfd3a19eebaa352d3409a7ecb691a63d5a0c470ff32cf3bb7c42f63d"} Jul 06 20:19:08.145880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:08.145833 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7w72c" podStartSLOduration=2.145817756 podStartE2EDuration="2.145817756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:08.14389311 +0000 UTC m=+264.634516661" watchObservedRunningTime="2026-07-06 20:19:08.145817756 +0000 UTC m=+264.636441305" Jul 06 20:19:10.122194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:10.122164 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:10.223198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:10.223170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:11.969427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:11.969401 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:12.407674 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.407647 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn"] Jul 06 20:19:12.409934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.409919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.412476 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.412449 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 20:19:12.413150 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.413122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nx8nz\"" Jul 06 20:19:12.413253 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.413130 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 20:19:12.421967 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.421951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn"] Jul 06 20:19:12.494419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.494398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.494514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.494431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.494514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.494456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnwt\" (UniqueName: \"kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.595689 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.595665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.595779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.595702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnwt\" (UniqueName: \"kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.595779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.595743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.596113 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.596094 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.596113 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.596106 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.604212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.604194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnwt\" (UniqueName: \"kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.718843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.718800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:12.840303 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:12.840231 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn"] Jul 06 20:19:12.842282 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:12.842261 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e39914a_48e4_4f1e_a471_8634381871d4.slice/crio-62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc WatchSource:0}: Error finding container 62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc: Status 404 returned error can't find the container with id 62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc Jul 06 20:19:13.139483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:13.139415 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e39914a-48e4-4f1e-a471-8634381871d4" containerID="9ee9f789d4c51b0062d83267fda1c49c2897f529c01fb459371234736aaefa30" exitCode=0 Jul 06 20:19:13.139822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:13.139502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" event={"ID":"2e39914a-48e4-4f1e-a471-8634381871d4","Type":"ContainerDied","Data":"9ee9f789d4c51b0062d83267fda1c49c2897f529c01fb459371234736aaefa30"} Jul 06 20:19:13.139822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:13.139560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" event={"ID":"2e39914a-48e4-4f1e-a471-8634381871d4","Type":"ContainerStarted","Data":"62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc"} Jul 06 20:19:14.144457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:14.144427 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e39914a-48e4-4f1e-a471-8634381871d4" containerID="90f05c923797813876d1afcfe88596d25b3edd17660e488a39ad676ff4d68e00" exitCode=0 Jul 06 20:19:14.144809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:14.144462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" event={"ID":"2e39914a-48e4-4f1e-a471-8634381871d4","Type":"ContainerDied","Data":"90f05c923797813876d1afcfe88596d25b3edd17660e488a39ad676ff4d68e00"} Jul 06 20:19:15.150671 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:15.150634 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e39914a-48e4-4f1e-a471-8634381871d4" containerID="138a9b93c7913d1c8f078bb7fe73bd0699b81adfc1fcb6d92a46284175d9d969" exitCode=0 Jul 06 20:19:15.151057 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:15.150671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" event={"ID":"2e39914a-48e4-4f1e-a471-8634381871d4","Type":"ContainerDied","Data":"138a9b93c7913d1c8f078bb7fe73bd0699b81adfc1fcb6d92a46284175d9d969"} Jul 06 20:19:16.274723 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.274700 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:16.325480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.325454 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle\") pod \"2e39914a-48e4-4f1e-a471-8634381871d4\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " Jul 06 20:19:16.325585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.325491 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util\") pod \"2e39914a-48e4-4f1e-a471-8634381871d4\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " Jul 06 20:19:16.325585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.325514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngnwt\" (UniqueName: \"kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt\") pod \"2e39914a-48e4-4f1e-a471-8634381871d4\" (UID: \"2e39914a-48e4-4f1e-a471-8634381871d4\") " Jul 06 20:19:16.326220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.326196 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle" (OuterVolumeSpecName: "bundle") pod "2e39914a-48e4-4f1e-a471-8634381871d4" (UID: "2e39914a-48e4-4f1e-a471-8634381871d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:19:16.327508 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.327487 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt" (OuterVolumeSpecName: "kube-api-access-ngnwt") pod "2e39914a-48e4-4f1e-a471-8634381871d4" (UID: "2e39914a-48e4-4f1e-a471-8634381871d4"). InnerVolumeSpecName "kube-api-access-ngnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:19:16.331368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.331337 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util" (OuterVolumeSpecName: "util") pod "2e39914a-48e4-4f1e-a471-8634381871d4" (UID: "2e39914a-48e4-4f1e-a471-8634381871d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:19:16.425979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.425955 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:16.425979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.425975 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e39914a-48e4-4f1e-a471-8634381871d4-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:16.426098 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:16.425984 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnwt\" (UniqueName: \"kubernetes.io/projected/2e39914a-48e4-4f1e-a471-8634381871d4-kube-api-access-ngnwt\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:17.160507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:17.160476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" event={"ID":"2e39914a-48e4-4f1e-a471-8634381871d4","Type":"ContainerDied","Data":"62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc"} Jul 06 20:19:17.160507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:17.160493 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affblvn" Jul 06 20:19:17.160679 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:17.160514 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62904a540e73483670bee4e7a5b859f6b8384f7b505d4e8bdcfa16c5130f0dcc" Jul 06 20:19:22.664552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:22.664516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:22.765024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:22.764995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:24.084738 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:24.084706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:24.187246 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:24.187218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:24.701650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:24.701624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:24.992053 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:24.991989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.099068 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.099041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.118016 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.117985 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf"] Jul 06 20:19:25.118231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118220 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="pull" Jul 06 20:19:25.118277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118233 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="pull" Jul 06 20:19:25.118277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118248 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="util" Jul 06 20:19:25.118277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118254 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="util" Jul 06 20:19:25.118277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118261 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="extract" Jul 06 20:19:25.118277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118266 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="extract" Jul 06 20:19:25.118417 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.118313 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e39914a-48e4-4f1e-a471-8634381871d4" containerName="extract" Jul 06 20:19:25.123175 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.123159 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.126647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.126626 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 20:19:25.126806 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.126787 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-88kxf\"" Jul 06 20:19:25.127155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.127138 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 20:19:25.127252 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.127213 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 20:19:25.127252 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.127243 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 20:19:25.143547 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.143526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf"] Jul 06 20:19:25.182197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.182175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjhr\" (UniqueName: \"kubernetes.io/projected/2b1a752a-c485-4755-9cfd-9a4ec86691f8-kube-api-access-vxjhr\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.182275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.182207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-manifests\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.182275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.182235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.182352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.182309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.182389 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.182348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-tmp\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.258899 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.258844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.282980 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.282959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.283049 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.282987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.283049 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.283006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-tmp\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.283116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.283056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjhr\" (UniqueName: \"kubernetes.io/projected/2b1a752a-c485-4755-9cfd-9a4ec86691f8-kube-api-access-vxjhr\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.283116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.283093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-manifests\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.283525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.283508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-manifests\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.285285 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.285258 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b1a752a-c485-4755-9cfd-9a4ec86691f8-tmp\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.285499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.285482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.285499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.285494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b1a752a-c485-4755-9cfd-9a4ec86691f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.292130 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.292109 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjhr\" (UniqueName: \"kubernetes.io/projected/2b1a752a-c485-4755-9cfd-9a4ec86691f8-kube-api-access-vxjhr\") pod \"opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf\" (UID: \"2b1a752a-c485-4755-9cfd-9a4ec86691f8\") " pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.356285 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.356245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.433925 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.433677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:25.457514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.457485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.585264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.585218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.592602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.592576 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf"] Jul 06 20:19:25.596754 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:25.596724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1a752a_c485_4755_9cfd_9a4ec86691f8.slice/crio-044944e4ed061437f093a222fb1a598ab0f39ee228ce38323fa71fa581997bee WatchSource:0}: Error finding container 044944e4ed061437f093a222fb1a598ab0f39ee228ce38323fa71fa581997bee: Status 404 returned error can't find the container with id 044944e4ed061437f093a222fb1a598ab0f39ee228ce38323fa71fa581997bee Jul 06 20:19:25.683430 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.683400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.784212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.784136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:25.887719 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:25.887685 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:26.195034 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:26.195005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" event={"ID":"2b1a752a-c485-4755-9cfd-9a4ec86691f8","Type":"ContainerStarted","Data":"044944e4ed061437f093a222fb1a598ab0f39ee228ce38323fa71fa581997bee"} Jul 06 20:19:29.210699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:29.210664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" event={"ID":"2b1a752a-c485-4755-9cfd-9a4ec86691f8","Type":"ContainerStarted","Data":"d225cd16082ab9a793050bc7524cf27d55be0235bc66550d931f1fb696f3dec1"} Jul 06 20:19:30.217706 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:30.217675 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b1a752a-c485-4755-9cfd-9a4ec86691f8" containerID="d225cd16082ab9a793050bc7524cf27d55be0235bc66550d931f1fb696f3dec1" exitCode=0 Jul 06 20:19:30.218086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:30.217760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" event={"ID":"2b1a752a-c485-4755-9cfd-9a4ec86691f8","Type":"ContainerDied","Data":"d225cd16082ab9a793050bc7524cf27d55be0235bc66550d931f1fb696f3dec1"} Jul 06 20:19:31.224556 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:31.224522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" event={"ID":"2b1a752a-c485-4755-9cfd-9a4ec86691f8","Type":"ContainerStarted","Data":"62b17a802b45efaa7096cb4e411e2b91a6e03070429316331a17bfcbffb2e214"} Jul 06 20:19:31.225060 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:31.224656 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:31.246721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:31.246674 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" podStartSLOduration=1.7894487589999999 podStartE2EDuration="6.246660225s" totalImagesPullingTime="3.453418804s" totalInitContainerRuntime="1.003792662s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:31.245752117 +0000 UTC m=+287.736375667" watchObservedRunningTime="2026-07-06 20:19:31.246660225 +0000 UTC m=+287.737283774" Jul 06 20:19:35.615777 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:35.615735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:35.989830 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:35.989798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:36.090391 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:36.090362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:39.968554 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:39.968520 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:40.074543 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:40.074511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:42.229662 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:42.229632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf" Jul 06 20:19:43.910161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:43.910132 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:19:43.910715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:43.910691 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:19:43.918269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:43.918253 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 20:19:44.205778 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.205694 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:44.360172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.360130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:44.868826 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.868795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:44.875028 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.875003 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d"] Jul 06 20:19:44.878113 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.878097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:44.884333 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.884317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 20:19:44.885190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.885175 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 20:19:44.886238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.886218 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 20:19:44.886527 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.886514 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 20:19:44.899592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:44.899573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d"] Jul 06 20:19:45.022572 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.022544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tls-certs\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.022931 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.022594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tmp\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.022931 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.022643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-kube-api-access-nkjkf\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.059321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.059291 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rmsfb"] Jul 06 20:19:45.063585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.063564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.068086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.068065 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 20:19:45.068086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.068088 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rdw8r\"" Jul 06 20:19:45.069202 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.069180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:45.083715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.083686 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rmsfb"] Jul 06 20:19:45.123019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.122954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.123019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.122987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tls-certs\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.123172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.123052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tmp\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.123172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.123108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9mk\" (UniqueName: \"kubernetes.io/projected/85d53d1d-83cb-4a48-a038-0848e25232f3-kube-api-access-xk9mk\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.123172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.123129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-kube-api-access-nkjkf\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.125073 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.125056 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tmp\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.125277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.125261 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-tls-certs\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.134897 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.134874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/d3c53cfd-f962-4ef1-8de9-ba591d9e49e1-kube-api-access-nkjkf\") pod \"kube-auth-proxy-7f9c87b66f-5kn9d\" (UID: \"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1\") " pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.162441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.162422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:45.187238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.187212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" Jul 06 20:19:45.224176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.224149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9mk\" (UniqueName: \"kubernetes.io/projected/85d53d1d-83cb-4a48-a038-0848e25232f3-kube-api-access-xk9mk\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.224283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.224188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.224283 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:45.224280 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 20:19:45.224375 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:45.224327 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert podName:85d53d1d-83cb-4a48-a038-0848e25232f3 nodeName:}" failed. No retries permitted until 2026-07-06 20:19:45.724313772 +0000 UTC m=+302.214937300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert") pod "odh-model-controller-7998c67dc4-rmsfb" (UID: "85d53d1d-83cb-4a48-a038-0848e25232f3") : secret "odh-model-controller-webhook-cert" not found Jul 06 20:19:45.238442 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.238401 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9mk\" (UniqueName: \"kubernetes.io/projected/85d53d1d-83cb-4a48-a038-0848e25232f3-kube-api-access-xk9mk\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.303835 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.303810 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:45.336949 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.336928 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d"] Jul 06 20:19:45.339217 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:45.339197 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c53cfd_f962_4ef1_8de9_ba591d9e49e1.slice/crio-f6e6d80a4cd23f71f2c159227d2a5b0c54d5854a168648939384e4c7d9c66975 WatchSource:0}: Error finding container f6e6d80a4cd23f71f2c159227d2a5b0c54d5854a168648939384e4c7d9c66975: Status 404 returned error can't find the container with id f6e6d80a4cd23f71f2c159227d2a5b0c54d5854a168648939384e4c7d9c66975 Jul 06 20:19:45.340925 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.340891 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:19:45.612565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.612535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:45.728463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.728432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.730932 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.730886 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d53d1d-83cb-4a48-a038-0848e25232f3-cert\") pod \"odh-model-controller-7998c67dc4-rmsfb\" (UID: \"85d53d1d-83cb-4a48-a038-0848e25232f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.821296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.821265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:45.948144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.948111 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl4zh"] Jul 06 20:19:45.952578 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.952559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:45.957848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.957830 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 20:19:45.958707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.958685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-44csc\"" Jul 06 20:19:45.973532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.973488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:45.979101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:45.979079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl4zh"] Jul 06 20:19:46.030526 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.030494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7r5\" (UniqueName: \"kubernetes.io/projected/19494732-31c5-43ef-a501-d15af99b6892-kube-api-access-rk7r5\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.030944 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.030550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19494732-31c5-43ef-a501-d15af99b6892-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.093235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.093192 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:46.127411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.127383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rmsfb"] Jul 06 20:19:46.131176 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:46.131146 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d53d1d_83cb_4a48_a038_0848e25232f3.slice/crio-ea20329af601e9430a46e31427438a6078ae66b4c5616cd8c2f078e856d581d6 WatchSource:0}: Error finding container ea20329af601e9430a46e31427438a6078ae66b4c5616cd8c2f078e856d581d6: Status 404 returned error can't find the container with id ea20329af601e9430a46e31427438a6078ae66b4c5616cd8c2f078e856d581d6 Jul 06 20:19:46.131414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.131394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7r5\" (UniqueName: \"kubernetes.io/projected/19494732-31c5-43ef-a501-d15af99b6892-kube-api-access-rk7r5\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.131508 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.131442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19494732-31c5-43ef-a501-d15af99b6892-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.134426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.134392 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19494732-31c5-43ef-a501-d15af99b6892-tls-certs\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.141349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.141320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7r5\" (UniqueName: \"kubernetes.io/projected/19494732-31c5-43ef-a501-d15af99b6892-kube-api-access-rk7r5\") pod \"model-serving-api-54d89bd5fc-sl4zh\" (UID: \"19494732-31c5-43ef-a501-d15af99b6892\") " pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.263006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.262916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:46.287803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.287760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" event={"ID":"85d53d1d-83cb-4a48-a038-0848e25232f3","Type":"ContainerStarted","Data":"ea20329af601e9430a46e31427438a6078ae66b4c5616cd8c2f078e856d581d6"} Jul 06 20:19:46.288881 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.288855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" event={"ID":"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1","Type":"ContainerStarted","Data":"f6e6d80a4cd23f71f2c159227d2a5b0c54d5854a168648939384e4c7d9c66975"} Jul 06 20:19:46.396623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.396596 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:46.400240 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.400221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.402722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.402666 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 20:19:46.402722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.402713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j5z9h\"" Jul 06 20:19:46.403015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.403001 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 20:19:46.406029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.406005 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 20:19:46.406283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.406263 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 20:19:46.409515 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.409491 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:46.413742 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.413722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:46.424475 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.424455 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sl4zh"] Jul 06 20:19:46.429388 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:46.429361 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19494732_31c5_43ef_a501_d15af99b6892.slice/crio-80e2a26bdb50b982db4a02ab4a7dcf0756a55437392448f3f1dd655f2ee68b03 WatchSource:0}: Error finding container 80e2a26bdb50b982db4a02ab4a7dcf0756a55437392448f3f1dd655f2ee68b03: Status 404 returned error can't find the container with id 80e2a26bdb50b982db4a02ab4a7dcf0756a55437392448f3f1dd655f2ee68b03 Jul 06 20:19:46.535203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrxr\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.535468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.535452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.548312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.548277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:46.636658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrxr\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.636833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.636833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.637854 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.637788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.639450 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.639429 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.639528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.639459 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.639600 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.639583 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.639694 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.639673 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.649241 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.649218 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.652992 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.652968 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrxr\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr\") pod \"istiod-openshift-gateway-7574459656-b4wcv\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.711801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.711778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:46.864676 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:46.864599 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:46.869135 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:46.869096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf294b96_435e_469c_8fd0_51d14ad7ab26.slice/crio-85f7472d0978db69ed7a8d0b6a2f3b7ae7d12fdabe9f71bc29f1fbed6465fec1 WatchSource:0}: Error finding container 85f7472d0978db69ed7a8d0b6a2f3b7ae7d12fdabe9f71bc29f1fbed6465fec1: Status 404 returned error can't find the container with id 85f7472d0978db69ed7a8d0b6a2f3b7ae7d12fdabe9f71bc29f1fbed6465fec1 Jul 06 20:19:47.296698 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:47.296622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" event={"ID":"19494732-31c5-43ef-a501-d15af99b6892","Type":"ContainerStarted","Data":"80e2a26bdb50b982db4a02ab4a7dcf0756a55437392448f3f1dd655f2ee68b03"} Jul 06 20:19:47.298653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:47.298617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" event={"ID":"cf294b96-435e-469c-8fd0-51d14ad7ab26","Type":"ContainerStarted","Data":"85f7472d0978db69ed7a8d0b6a2f3b7ae7d12fdabe9f71bc29f1fbed6465fec1"} Jul 06 20:19:47.677048 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:47.676959 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:50.314310 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.314276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" event={"ID":"d3c53cfd-f962-4ef1-8de9-ba591d9e49e1","Type":"ContainerStarted","Data":"cd21e0f4a2166ef0c9957d8f448ab65c14bbc8997a2e6da1e6baa7cf68ff1ffb"} Jul 06 20:19:50.315791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.315755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" event={"ID":"19494732-31c5-43ef-a501-d15af99b6892","Type":"ContainerStarted","Data":"4e9bf350a00b86909e0f13b6bf668ced7ecb682e638663a07b5e06b626e621ed"} Jul 06 20:19:50.315943 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.315849 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:19:50.317215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.317191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" event={"ID":"85d53d1d-83cb-4a48-a038-0848e25232f3","Type":"ContainerStarted","Data":"665d88d467d7e5a47dac15b1e0e19f6303754630b067252a109a35075693a2b4"} Jul 06 20:19:50.317334 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.317245 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:19:50.363238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.363191 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f9c87b66f-5kn9d" podStartSLOduration=1.563927403 podStartE2EDuration="6.363173308s" totalImagesPullingTime="4.799245905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:50.33356997 +0000 UTC m=+306.824193520" watchObservedRunningTime="2026-07-06 20:19:50.363173308 +0000 UTC m=+306.853796858" Jul 06 20:19:50.365034 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.364997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" podStartSLOduration=1.65662942 podStartE2EDuration="5.364983062s" totalImagesPullingTime="3.708353642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:50.361722394 +0000 UTC m=+306.852345945" watchObservedRunningTime="2026-07-06 20:19:50.364983062 +0000 UTC m=+306.855606614" Jul 06 20:19:50.382946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:50.382890 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" podStartSLOduration=1.318622474 podStartE2EDuration="5.382875739s" totalImagesPullingTime="4.064253265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:50.380914802 +0000 UTC m=+306.871538344" watchObservedRunningTime="2026-07-06 20:19:50.382875739 +0000 UTC m=+306.873499290" Jul 06 20:19:52.975438 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:52.975395 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:19:52.975700 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:52.975461 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:19:53.334848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:53.334765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" event={"ID":"cf294b96-435e-469c-8fd0-51d14ad7ab26","Type":"ContainerStarted","Data":"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275"} Jul 06 20:19:53.335003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:53.334885 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:53.362624 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:53.362581 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" podStartSLOduration=1.259005334 podStartE2EDuration="7.362567909s" totalImagesPullingTime="6.103562575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:53.360163412 +0000 UTC m=+309.850786961" watchObservedRunningTime="2026-07-06 20:19:53.362567909 +0000 UTC m=+309.853191459" Jul 06 20:19:53.946390 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:53.946351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:54.069806 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:54.069770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:54.340657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:54.340585 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b4wcv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:19:54.340657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:54.340634 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:19:55.176767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.176738 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p"] Jul 06 20:19:55.179840 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.179823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.183105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.183089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b22jg\"" Jul 06 20:19:55.208917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.202540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p"] Jul 06 20:19:55.310612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310730 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310730 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthns\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-kube-api-access-hthns\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.310863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.311073 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.311073 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.310889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.334719 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.334692 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:55.412225 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthns\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-kube-api-access-hthns\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412754 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.412813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.412789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.413037 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.413010 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.413186 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.413165 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.413321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.413302 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.414819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.414793 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.415400 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.415378 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.443138 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.442990 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.444585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.444413 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthns\" (UniqueName: \"kubernetes.io/projected/2ca16ad5-466a-4cb5-8d3b-df72530c6e57-kube-api-access-hthns\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k5z2p\" (UID: \"2ca16ad5-466a-4cb5-8d3b-df72530c6e57\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.481650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.481613 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:55.489875 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.489850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:55.649061 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.649032 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj"] Jul 06 20:19:55.653561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.653540 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn"] Jul 06 20:19:55.653695 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.653667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.656801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.656783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.658000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.657981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wc6hg\"" Jul 06 20:19:55.659215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.659197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 20:19:55.665248 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.665230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-btvtp\"" Jul 06 20:19:55.666979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.666963 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 20:19:55.671227 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.671204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:55.695289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.694789 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj"] Jul 06 20:19:55.696436 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.696397 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn"] Jul 06 20:19:55.697140 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:55.697102 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca16ad5_466a_4cb5_8d3b_df72530c6e57.slice/crio-f674c1c12265952afe4e61a7126e7b94920700989fcab3dd6b91d2e47a18d098 WatchSource:0}: Error finding container f674c1c12265952afe4e61a7126e7b94920700989fcab3dd6b91d2e47a18d098: Status 404 returned error can't find the container with id f674c1c12265952afe4e61a7126e7b94920700989fcab3dd6b91d2e47a18d098 Jul 06 20:19:55.698821 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.698797 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p"] Jul 06 20:19:55.804696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.804669 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:55.815871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.815843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbht\" (UniqueName: \"kubernetes.io/projected/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-kube-api-access-pwbht\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.816008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.815875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.816008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.815892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35369594-ea72-4a81-a41c-7298f7f73965-cert\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.816083 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.816045 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd64s\" (UniqueName: \"kubernetes.io/projected/35369594-ea72-4a81-a41c-7298f7f73965-kube-api-access-nd64s\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.916950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.916898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd64s\" (UniqueName: \"kubernetes.io/projected/35369594-ea72-4a81-a41c-7298f7f73965-kube-api-access-nd64s\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.917076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.916964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbht\" (UniqueName: \"kubernetes.io/projected/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-kube-api-access-pwbht\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.917076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.916984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.917076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.917002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35369594-ea72-4a81-a41c-7298f7f73965-cert\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.917237 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:55.917115 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:19:55.917237 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:55.917178 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert podName:4b7e81c5-75bc-47c8-b95a-7b3d13aedd24 nodeName:}" failed. No retries permitted until 2026-07-06 20:19:56.41715987 +0000 UTC m=+312.907783408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert") pod "llmisvc-controller-manager-777c6bd7d5-wpkdn" (UID: "4b7e81c5-75bc-47c8-b95a-7b3d13aedd24") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:19:55.919215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.919194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35369594-ea72-4a81-a41c-7298f7f73965-cert\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.935813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.935792 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd64s\" (UniqueName: \"kubernetes.io/projected/35369594-ea72-4a81-a41c-7298f7f73965-kube-api-access-nd64s\") pod \"kserve-controller-manager-d6f4b65b8-6fnqj\" (UID: \"35369594-ea72-4a81-a41c-7298f7f73965\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:55.966132 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.966077 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbht\" (UniqueName: \"kubernetes.io/projected/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-kube-api-access-pwbht\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:55.966208 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:55.966156 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:56.113774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.112553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj"] Jul 06 20:19:56.117160 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:56.117134 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35369594_ea72_4a81_a41c_7298f7f73965.slice/crio-a0b9b5ddcb82354e75e66129ab27ce9aabe4bbd932023ee3fb12a1b72e624159 WatchSource:0}: Error finding container a0b9b5ddcb82354e75e66129ab27ce9aabe4bbd932023ee3fb12a1b72e624159: Status 404 returned error can't find the container with id a0b9b5ddcb82354e75e66129ab27ce9aabe4bbd932023ee3fb12a1b72e624159 Jul 06 20:19:56.350260 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.350178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" event={"ID":"35369594-ea72-4a81-a41c-7298f7f73965","Type":"ContainerStarted","Data":"a0b9b5ddcb82354e75e66129ab27ce9aabe4bbd932023ee3fb12a1b72e624159"} Jul 06 20:19:56.351525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.351492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" event={"ID":"2ca16ad5-466a-4cb5-8d3b-df72530c6e57","Type":"ContainerStarted","Data":"f674c1c12265952afe4e61a7126e7b94920700989fcab3dd6b91d2e47a18d098"} Jul 06 20:19:56.420697 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.420672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:56.420826 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:56.420808 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 20:19:56.420885 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:56.420875 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert podName:4b7e81c5-75bc-47c8-b95a-7b3d13aedd24 nodeName:}" failed. No retries permitted until 2026-07-06 20:19:57.420860571 +0000 UTC m=+313.911484103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert") pod "llmisvc-controller-manager-777c6bd7d5-wpkdn" (UID: "4b7e81c5-75bc-47c8-b95a-7b3d13aedd24") : secret "llmisvc-webhook-server-cert" not found Jul 06 20:19:56.778598 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.778562 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:56.892409 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.892372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:56.995222 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:56.995173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:57.340221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.340179 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:57.430576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.430121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:57.432994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.432970 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b7e81c5-75bc-47c8-b95a-7b3d13aedd24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wpkdn\" (UID: \"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:57.473047 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.473018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:19:57.769569 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.769526 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:57.769779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:57.769754 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerName="discovery" containerID="cri-o://8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275" gracePeriod=30 Jul 06 20:19:58.498387 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.498336 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:19:58.498843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.498434 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:19:58.498843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.498474 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:19:58.580711 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.580682 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn"] Jul 06 20:19:58.584436 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:19:58.584407 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b7e81c5_75bc_47c8_b95a_7b3d13aedd24.slice/crio-fce9a460b6ae7e0038ceec22ef769121bc320a5ee3061ca375f0fe4c5990287c WatchSource:0}: Error finding container fce9a460b6ae7e0038ceec22ef769121bc320a5ee3061ca375f0fe4c5990287c: Status 404 returned error can't find the container with id fce9a460b6ae7e0038ceec22ef769121bc320a5ee3061ca375f0fe4c5990287c Jul 06 20:19:58.591720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.591699 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:58.593304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.593279 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:58.727172 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.727096 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:19:58.748763 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748741 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.748873 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.748873 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxrxr\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.748997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748896 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.748997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748961 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.748997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.748986 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.749153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.749038 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts\") pod \"cf294b96-435e-469c-8fd0-51d14ad7ab26\" (UID: \"cf294b96-435e-469c-8fd0-51d14ad7ab26\") " Jul 06 20:19:58.750038 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.749991 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:19:58.751649 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.751605 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:19:58.752378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.752337 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:19:58.752775 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.752722 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs" (OuterVolumeSpecName: "local-certs") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:19:58.752894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.752859 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts" (OuterVolumeSpecName: "cacerts") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:19:58.753203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.753183 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr" (OuterVolumeSpecName: "kube-api-access-vxrxr") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "kube-api-access-vxrxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:19:58.753370 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.753280 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cf294b96-435e-469c-8fd0-51d14ad7ab26" (UID: "cf294b96-435e-469c-8fd0-51d14ad7ab26"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:19:58.850102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850070 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf294b96-435e-469c-8fd0-51d14ad7ab26-local-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850103 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-ca-configmap\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850116 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-cacerts\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850128 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850142 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-kubeconfig\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850152 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrxr\" (UniqueName: \"kubernetes.io/projected/cf294b96-435e-469c-8fd0-51d14ad7ab26-kube-api-access-vxrxr\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:58.850278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:58.850164 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cf294b96-435e-469c-8fd0-51d14ad7ab26-istio-csr-dns-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:19:59.368067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.368038 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerID="8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275" exitCode=0 Jul 06 20:19:59.368161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.368074 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" Jul 06 20:19:59.368161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.368107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" event={"ID":"cf294b96-435e-469c-8fd0-51d14ad7ab26","Type":"ContainerDied","Data":"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275"} Jul 06 20:19:59.368161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.368154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv" event={"ID":"cf294b96-435e-469c-8fd0-51d14ad7ab26","Type":"ContainerDied","Data":"85f7472d0978db69ed7a8d0b6a2f3b7ae7d12fdabe9f71bc29f1fbed6465fec1"} Jul 06 20:19:59.368293 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.368177 2574 scope.go:117] "RemoveContainer" containerID="8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275" Jul 06 20:19:59.369741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.369715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" event={"ID":"35369594-ea72-4a81-a41c-7298f7f73965","Type":"ContainerStarted","Data":"3b3528415a88be8b8dc16bfb60eb6b7deb09794b24516f62fc6507a144ec59c8"} Jul 06 20:19:59.369859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.369784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:19:59.372080 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.371602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" event={"ID":"2ca16ad5-466a-4cb5-8d3b-df72530c6e57","Type":"ContainerStarted","Data":"b0b504d9dd2fa9d92e36fa6440694d5eb6e47b54eccb5578970d033cc2aeb725"} Jul 06 20:19:59.373023 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.373000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" event={"ID":"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24","Type":"ContainerStarted","Data":"fce9a460b6ae7e0038ceec22ef769121bc320a5ee3061ca375f0fe4c5990287c"} Jul 06 20:19:59.419816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.419760 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" podStartSLOduration=1.621496579 podStartE2EDuration="4.419744304s" totalImagesPullingTime="2.798247725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:59.417230203 +0000 UTC m=+315.907853768" watchObservedRunningTime="2026-07-06 20:19:59.419744304 +0000 UTC m=+315.910367854" Jul 06 20:19:59.424750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.424722 2574 scope.go:117] "RemoveContainer" containerID="8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275" Jul 06 20:19:59.425110 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:19:59.425086 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275\": container with ID starting with 8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275 not found: ID does not exist" containerID="8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275" Jul 06 20:19:59.425199 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.425117 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275"} err="failed to get container status \"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275\": rpc error: code = NotFound desc = could not find container \"8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275\": container with ID starting with 8d4167436636d0ef9334650fbef5f325ba05476fa2945e69a96ee216a583c275 not found: ID does not exist" Jul 06 20:19:59.446102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.446060 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" podStartSLOduration=1.258059536 podStartE2EDuration="4.446045792s" totalImagesPullingTime="3.187986256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:59.443742431 +0000 UTC m=+315.934366030" watchObservedRunningTime="2026-07-06 20:19:59.446045792 +0000 UTC m=+315.936669341" Jul 06 20:19:59.476884 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.476860 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:59.483583 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.483557 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b4wcv"] Jul 06 20:19:59.490623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.490597 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:19:59.495489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:19:59.495444 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:20:00.072226 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:00.072192 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" path="/var/lib/kubelet/pods/cf294b96-435e-469c-8fd0-51d14ad7ab26/volumes" Jul 06 20:20:00.378634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:00.378561 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:20:00.379406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:00.379389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k5z2p" Jul 06 20:20:01.324989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:01.324960 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rmsfb" Jul 06 20:20:01.326994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:01.326971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sl4zh" Jul 06 20:20:03.394882 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:03.394850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" event={"ID":"4b7e81c5-75bc-47c8-b95a-7b3d13aedd24","Type":"ContainerStarted","Data":"6acb6e84714b8124360b01c07ad987e794e6640b1c5a28c30794a101826cfc5c"} Jul 06 20:20:03.395308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:03.394932 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:20:03.413265 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:03.413224 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" podStartSLOduration=4.641527196 podStartE2EDuration="8.413210858s" totalImagesPullingTime="3.771683662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:20:03.410864162 +0000 UTC m=+319.901487716" watchObservedRunningTime="2026-07-06 20:20:03.413210858 +0000 UTC m=+319.903834407" Jul 06 20:20:30.383538 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:30.383508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6fnqj" Jul 06 20:20:34.401981 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:34.401948 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wpkdn" Jul 06 20:20:34.736983 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:34.736956 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:20:45.684470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.684428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:20:45.783465 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.783429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:20:45.808492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.808465 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:45.808819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.808804 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerName="discovery" Jul 06 20:20:45.808891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.808822 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerName="discovery" Jul 06 20:20:45.808984 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.808925 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf294b96-435e-469c-8fd0-51d14ad7ab26" containerName="discovery" Jul 06 20:20:45.811868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.811849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:45.814234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.814214 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 20:20:45.815085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.815065 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 20:20:45.815085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.815077 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7vsl9\"" Jul 06 20:20:45.821290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.821266 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:45.869644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.869625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:20:45.888822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.888801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9df9\" (UniqueName: \"kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9\") pod \"kuadrant-operator-catalog-6zf6s\" (UID: \"b9eb352d-c68b-493f-9b3a-3442502b69dc\") " pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:45.989435 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.989382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9df9\" (UniqueName: \"kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9\") pod \"kuadrant-operator-catalog-6zf6s\" (UID: \"b9eb352d-c68b-493f-9b3a-3442502b69dc\") " pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:45.997099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:45.997082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9df9\" (UniqueName: \"kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9\") pod \"kuadrant-operator-catalog-6zf6s\" (UID: \"b9eb352d-c68b-493f-9b3a-3442502b69dc\") " pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:46.122701 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.122669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:46.175877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.175849 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:46.245168 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.245145 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:46.383498 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.383463 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vfrmm"] Jul 06 20:20:46.386140 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.386117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:46.395162 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.395141 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vfrmm"] Jul 06 20:20:46.493706 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.493679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h9q\" (UniqueName: \"kubernetes.io/projected/b3c43d88-0d0a-4647-870f-7550e4e7f9c5-kube-api-access-28h9q\") pod \"kuadrant-operator-catalog-vfrmm\" (UID: \"b3c43d88-0d0a-4647-870f-7550e4e7f9c5\") " pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:46.589420 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.589353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" event={"ID":"b9eb352d-c68b-493f-9b3a-3442502b69dc","Type":"ContainerStarted","Data":"5194709abf25727e0ad399fcb7777af30d29f3a33557d7f5073a414dc6050668"} Jul 06 20:20:46.594630 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.594611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h9q\" (UniqueName: \"kubernetes.io/projected/b3c43d88-0d0a-4647-870f-7550e4e7f9c5-kube-api-access-28h9q\") pod \"kuadrant-operator-catalog-vfrmm\" (UID: \"b3c43d88-0d0a-4647-870f-7550e4e7f9c5\") " pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:46.602492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.602466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h9q\" (UniqueName: \"kubernetes.io/projected/b3c43d88-0d0a-4647-870f-7550e4e7f9c5-kube-api-access-28h9q\") pod \"kuadrant-operator-catalog-vfrmm\" (UID: \"b3c43d88-0d0a-4647-870f-7550e4e7f9c5\") " pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:46.696661 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.696636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:46.823450 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:46.823428 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vfrmm"] Jul 06 20:20:46.861317 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:20:46.861254 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c43d88_0d0a_4647_870f_7550e4e7f9c5.slice/crio-2b75056ed015dbf62a979f1312991112e65814559b4b839f6b9caa41937c00db WatchSource:0}: Error finding container 2b75056ed015dbf62a979f1312991112e65814559b4b839f6b9caa41937c00db: Status 404 returned error can't find the container with id 2b75056ed015dbf62a979f1312991112e65814559b4b839f6b9caa41937c00db Jul 06 20:20:47.596283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:47.596251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" event={"ID":"b3c43d88-0d0a-4647-870f-7550e4e7f9c5","Type":"ContainerStarted","Data":"2b75056ed015dbf62a979f1312991112e65814559b4b839f6b9caa41937c00db"} Jul 06 20:20:49.611754 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.611717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" event={"ID":"b3c43d88-0d0a-4647-870f-7550e4e7f9c5","Type":"ContainerStarted","Data":"afd7ab6f54c1c97dffa1d5395d0f090af7cdc9ba395c58cf4f5d9b15a2f8dc70"} Jul 06 20:20:49.613158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.613128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" event={"ID":"b9eb352d-c68b-493f-9b3a-3442502b69dc","Type":"ContainerStarted","Data":"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a"} Jul 06 20:20:49.613271 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.613186 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" podUID="b9eb352d-c68b-493f-9b3a-3442502b69dc" containerName="registry-server" containerID="cri-o://eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a" gracePeriod=2 Jul 06 20:20:49.627883 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.627845 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" podStartSLOduration=1.950602868 podStartE2EDuration="3.627830367s" totalImagesPullingTime="1.677227499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:20:49.626324837 +0000 UTC m=+366.116948392" watchObservedRunningTime="2026-07-06 20:20:49.627830367 +0000 UTC m=+366.118453920" Jul 06 20:20:49.643462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.643421 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" podStartSLOduration=2.356281182 podStartE2EDuration="4.64340859s" totalImagesPullingTime="2.287127408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:20:49.64245905 +0000 UTC m=+366.133082600" watchObservedRunningTime="2026-07-06 20:20:49.64340859 +0000 UTC m=+366.134032140" Jul 06 20:20:49.856876 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.856850 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:49.926162 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.926136 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9df9\" (UniqueName: \"kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9\") pod \"b9eb352d-c68b-493f-9b3a-3442502b69dc\" (UID: \"b9eb352d-c68b-493f-9b3a-3442502b69dc\") " Jul 06 20:20:49.928093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:49.928073 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9" (OuterVolumeSpecName: "kube-api-access-w9df9") pod "b9eb352d-c68b-493f-9b3a-3442502b69dc" (UID: "b9eb352d-c68b-493f-9b3a-3442502b69dc"). InnerVolumeSpecName "kube-api-access-w9df9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:20:50.027299 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.027273 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9df9\" (UniqueName: \"kubernetes.io/projected/b9eb352d-c68b-493f-9b3a-3442502b69dc-kube-api-access-w9df9\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:20:50.619457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.619415 2574 generic.go:356] "Generic (PLEG): container finished" podID="b9eb352d-c68b-493f-9b3a-3442502b69dc" containerID="eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a" exitCode=0 Jul 06 20:20:50.619457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.619454 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" Jul 06 20:20:50.619978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.619513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" event={"ID":"b9eb352d-c68b-493f-9b3a-3442502b69dc","Type":"ContainerDied","Data":"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a"} Jul 06 20:20:50.619978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.619548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6zf6s" event={"ID":"b9eb352d-c68b-493f-9b3a-3442502b69dc","Type":"ContainerDied","Data":"5194709abf25727e0ad399fcb7777af30d29f3a33557d7f5073a414dc6050668"} Jul 06 20:20:50.619978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.619566 2574 scope.go:117] "RemoveContainer" containerID="eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a" Jul 06 20:20:50.628086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.628073 2574 scope.go:117] "RemoveContainer" containerID="eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a" Jul 06 20:20:50.628347 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:20:50.628329 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a\": container with ID starting with eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a not found: ID does not exist" containerID="eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a" Jul 06 20:20:50.628405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.628357 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a"} err="failed to get container status \"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a\": rpc error: code = NotFound desc = could not find container \"eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a\": container with ID starting with eded6dc1f3b0299016cf7437434f5436e620d9b98e079bf0b6803ce5b9eb6b4a not found: ID does not exist" Jul 06 20:20:50.636677 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.636659 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:50.640733 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:50.640709 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6zf6s"] Jul 06 20:20:52.071433 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:52.071402 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9eb352d-c68b-493f-9b3a-3442502b69dc" path="/var/lib/kubelet/pods/b9eb352d-c68b-493f-9b3a-3442502b69dc/volumes" Jul 06 20:20:56.697735 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:56.697705 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:56.698312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:56.697783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:56.719871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:56.719849 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:20:57.671853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:20:57.671829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vfrmm" Jul 06 20:21:01.188943 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.188898 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv"] Jul 06 20:21:01.189325 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.189207 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9eb352d-c68b-493f-9b3a-3442502b69dc" containerName="registry-server" Jul 06 20:21:01.189325 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.189219 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9eb352d-c68b-493f-9b3a-3442502b69dc" containerName="registry-server" Jul 06 20:21:01.189325 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.189314 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9eb352d-c68b-493f-9b3a-3442502b69dc" containerName="registry-server" Jul 06 20:21:01.193834 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.193818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.196298 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.196276 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c29nn\"" Jul 06 20:21:01.201613 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.201590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv"] Jul 06 20:21:01.312675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.312652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.312805 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.312692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.312805 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.312730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvb8j\" (UniqueName: \"kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.413228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.413201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.413374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.413243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvb8j\" (UniqueName: \"kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.413374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.413289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.413573 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.413553 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.413612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.413566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.421501 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.421473 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvb8j\" (UniqueName: \"kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.504078 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.504029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:01.631607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.631581 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv"] Jul 06 20:21:01.634873 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:01.634844 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd88a946_654f_4e5d_a400_18d4b3a3ac3b.slice/crio-a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7 WatchSource:0}: Error finding container a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7: Status 404 returned error can't find the container with id a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7 Jul 06 20:21:01.668961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.668937 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" event={"ID":"dd88a946-654f-4e5d-a400-18d4b3a3ac3b","Type":"ContainerStarted","Data":"a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7"} Jul 06 20:21:01.794442 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.794378 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk"] Jul 06 20:21:01.797761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.797747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:01.806564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.806540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk"] Jul 06 20:21:01.918055 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.918023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:01.918209 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.918101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrz6\" (UniqueName: \"kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:01.918209 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:01.918125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.019314 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.019291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.019457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.019327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrz6\" (UniqueName: \"kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.019457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.019347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.019654 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.019636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.019693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.019685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.027577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.027549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrz6\" (UniqueName: \"kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.107113 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.107063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:02.198061 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.198033 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx"] Jul 06 20:21:02.203222 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.203196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.210268 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.210244 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx"] Jul 06 20:21:02.234639 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.234620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk"] Jul 06 20:21:02.238090 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:02.238069 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8942ef1e_616a_4fc1_949c_36de262b1c6f.slice/crio-bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a WatchSource:0}: Error finding container bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a: Status 404 returned error can't find the container with id bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a Jul 06 20:21:02.322580 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.322555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.322717 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.322618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdqg\" (UniqueName: \"kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.322784 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.322742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.424187 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.424161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.424287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.424214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdqg\" (UniqueName: \"kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.424287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.424265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.424591 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.424570 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.424634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.424574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.432124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.432105 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdqg\" (UniqueName: \"kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.515624 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.515603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:02.594939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.594899 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd"] Jul 06 20:21:02.600788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.600765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.608177 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.608156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd"] Jul 06 20:21:02.646400 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.646319 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx"] Jul 06 20:21:02.648487 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:02.648462 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4d62be_a38d_48c4_813e_eb6de01bff2a.slice/crio-ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc WatchSource:0}: Error finding container ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc: Status 404 returned error can't find the container with id ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc Jul 06 20:21:02.674668 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.674612 2574 generic.go:356] "Generic (PLEG): container finished" podID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerID="28df4258885524190e4ed0043bad233a55ee41b1d076b07392286df71fa7160e" exitCode=0 Jul 06 20:21:02.674752 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.674696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" event={"ID":"8942ef1e-616a-4fc1-949c-36de262b1c6f","Type":"ContainerDied","Data":"28df4258885524190e4ed0043bad233a55ee41b1d076b07392286df71fa7160e"} Jul 06 20:21:02.674813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.674757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" event={"ID":"8942ef1e-616a-4fc1-949c-36de262b1c6f","Type":"ContainerStarted","Data":"bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a"} Jul 06 20:21:02.681848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.681807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" event={"ID":"2b4d62be-a38d-48c4-813e-eb6de01bff2a","Type":"ContainerStarted","Data":"ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc"} Jul 06 20:21:02.683433 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.683409 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerID="8c9aee50f91d41f890d379a1a60c6824e22c9b2c8c8d7eb11b157bc615261b18" exitCode=0 Jul 06 20:21:02.683530 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.683468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" event={"ID":"dd88a946-654f-4e5d-a400-18d4b3a3ac3b","Type":"ContainerDied","Data":"8c9aee50f91d41f890d379a1a60c6824e22c9b2c8c8d7eb11b157bc615261b18"} Jul 06 20:21:02.725731 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.725711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7lv\" (UniqueName: \"kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.725808 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.725752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.725874 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.725830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.827301 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.827273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7lv\" (UniqueName: \"kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.827401 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.827322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.827557 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.827536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.827631 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.827606 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.827860 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.827840 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.835294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.835271 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7lv\" (UniqueName: \"kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:02.911990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:02.911961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:03.039062 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.039035 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd"] Jul 06 20:21:03.041301 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:03.041269 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2ea925_982a_4417_a43a_255d64b2e864.slice/crio-02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570 WatchSource:0}: Error finding container 02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570: Status 404 returned error can't find the container with id 02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570 Jul 06 20:21:03.689731 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.689696 2574 generic.go:356] "Generic (PLEG): container finished" podID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerID="ffa9e04e470146fb9eb3635ccf46a133fdffb22090609653f6745c007ca02b6f" exitCode=0 Jul 06 20:21:03.690085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.689783 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" event={"ID":"8942ef1e-616a-4fc1-949c-36de262b1c6f","Type":"ContainerDied","Data":"ffa9e04e470146fb9eb3635ccf46a133fdffb22090609653f6745c007ca02b6f"} Jul 06 20:21:03.691344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.691326 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerID="ce77a786ad4cb65c9ffbd0b6dd72244774b1a09038c5a018d2192768ac0db5a0" exitCode=0 Jul 06 20:21:03.691432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.691409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" event={"ID":"2b4d62be-a38d-48c4-813e-eb6de01bff2a","Type":"ContainerDied","Data":"ce77a786ad4cb65c9ffbd0b6dd72244774b1a09038c5a018d2192768ac0db5a0"} Jul 06 20:21:03.693350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.693314 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerID="309e0e712e61e2c00a0148b334ead242e249ed2ed867e114a4eb31502d537c85" exitCode=0 Jul 06 20:21:03.693506 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.693345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" event={"ID":"dd88a946-654f-4e5d-a400-18d4b3a3ac3b","Type":"ContainerDied","Data":"309e0e712e61e2c00a0148b334ead242e249ed2ed867e114a4eb31502d537c85"} Jul 06 20:21:03.695138 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.695119 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d2ea925-982a-4417-a43a-255d64b2e864" containerID="ac4225ba8728ab637d6259224046d9084d6ab3d7ffaf6f5557c44efee98130ba" exitCode=0 Jul 06 20:21:03.695215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.695186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" event={"ID":"2d2ea925-982a-4417-a43a-255d64b2e864","Type":"ContainerDied","Data":"ac4225ba8728ab637d6259224046d9084d6ab3d7ffaf6f5557c44efee98130ba"} Jul 06 20:21:03.695278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:03.695227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" event={"ID":"2d2ea925-982a-4417-a43a-255d64b2e864","Type":"ContainerStarted","Data":"02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570"} Jul 06 20:21:04.703942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.703884 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerID="2c47c7f0d9644079cf6dda54555dab8b03f17212aa8169ab5d99138c96dde229" exitCode=0 Jul 06 20:21:04.704269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.703948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" event={"ID":"dd88a946-654f-4e5d-a400-18d4b3a3ac3b","Type":"ContainerDied","Data":"2c47c7f0d9644079cf6dda54555dab8b03f17212aa8169ab5d99138c96dde229"} Jul 06 20:21:04.706087 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.706067 2574 generic.go:356] "Generic (PLEG): container finished" podID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerID="dc5744888bbdee847eee8b4d10f0cae710d58bde0fa932aa5880cf04340c96d5" exitCode=0 Jul 06 20:21:04.706179 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.706152 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" event={"ID":"8942ef1e-616a-4fc1-949c-36de262b1c6f","Type":"ContainerDied","Data":"dc5744888bbdee847eee8b4d10f0cae710d58bde0fa932aa5880cf04340c96d5"} Jul 06 20:21:04.707870 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.707848 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerID="60499296207e2ff52bc1f11f7996fe713507cc2fe7f032de09fa419bdc706da8" exitCode=0 Jul 06 20:21:04.707966 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:04.707882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" event={"ID":"2b4d62be-a38d-48c4-813e-eb6de01bff2a","Type":"ContainerDied","Data":"60499296207e2ff52bc1f11f7996fe713507cc2fe7f032de09fa419bdc706da8"} Jul 06 20:21:05.714823 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.714749 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d2ea925-982a-4417-a43a-255d64b2e864" containerID="9907d47d863feda59e2afc10100a315978493855bace6b9122ca5a05e5bd44ee" exitCode=0 Jul 06 20:21:05.715214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.714834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" event={"ID":"2d2ea925-982a-4417-a43a-255d64b2e864","Type":"ContainerDied","Data":"9907d47d863feda59e2afc10100a315978493855bace6b9122ca5a05e5bd44ee"} Jul 06 20:21:05.717140 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.717118 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerID="afa6f701e82b5abdf0ab4c0a708eaf7d5a2b5296c41355f3b25595f10624f54b" exitCode=0 Jul 06 20:21:05.717217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.717195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" event={"ID":"2b4d62be-a38d-48c4-813e-eb6de01bff2a","Type":"ContainerDied","Data":"afa6f701e82b5abdf0ab4c0a708eaf7d5a2b5296c41355f3b25595f10624f54b"} Jul 06 20:21:05.886790 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.886768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:05.891507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:05.891488 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:06.056857 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.056793 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle\") pod \"8942ef1e-616a-4fc1-949c-36de262b1c6f\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " Jul 06 20:21:06.056857 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.056850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbrz6\" (UniqueName: \"kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6\") pod \"8942ef1e-616a-4fc1-949c-36de262b1c6f\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " Jul 06 20:21:06.057033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.056879 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle\") pod \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " Jul 06 20:21:06.057033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.057016 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util\") pod \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " Jul 06 20:21:06.057101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.057053 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util\") pod \"8942ef1e-616a-4fc1-949c-36de262b1c6f\" (UID: \"8942ef1e-616a-4fc1-949c-36de262b1c6f\") " Jul 06 20:21:06.057101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.057076 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvb8j\" (UniqueName: \"kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j\") pod \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\" (UID: \"dd88a946-654f-4e5d-a400-18d4b3a3ac3b\") " Jul 06 20:21:06.057503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.057477 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle" (OuterVolumeSpecName: "bundle") pod "8942ef1e-616a-4fc1-949c-36de262b1c6f" (UID: "8942ef1e-616a-4fc1-949c-36de262b1c6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:06.057623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.057599 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle" (OuterVolumeSpecName: "bundle") pod "dd88a946-654f-4e5d-a400-18d4b3a3ac3b" (UID: "dd88a946-654f-4e5d-a400-18d4b3a3ac3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:06.059158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.059126 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j" (OuterVolumeSpecName: "kube-api-access-qvb8j") pod "dd88a946-654f-4e5d-a400-18d4b3a3ac3b" (UID: "dd88a946-654f-4e5d-a400-18d4b3a3ac3b"). InnerVolumeSpecName "kube-api-access-qvb8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:06.059253 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.059215 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6" (OuterVolumeSpecName: "kube-api-access-xbrz6") pod "8942ef1e-616a-4fc1-949c-36de262b1c6f" (UID: "8942ef1e-616a-4fc1-949c-36de262b1c6f"). InnerVolumeSpecName "kube-api-access-xbrz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:06.063202 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.063166 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util" (OuterVolumeSpecName: "util") pod "8942ef1e-616a-4fc1-949c-36de262b1c6f" (UID: "8942ef1e-616a-4fc1-949c-36de262b1c6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:06.063544 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.063526 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util" (OuterVolumeSpecName: "util") pod "dd88a946-654f-4e5d-a400-18d4b3a3ac3b" (UID: "dd88a946-654f-4e5d-a400-18d4b3a3ac3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:06.158402 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158382 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.158402 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158402 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.158537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158411 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvb8j\" (UniqueName: \"kubernetes.io/projected/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-kube-api-access-qvb8j\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.158537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158421 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8942ef1e-616a-4fc1-949c-36de262b1c6f-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.158537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158429 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrz6\" (UniqueName: \"kubernetes.io/projected/8942ef1e-616a-4fc1-949c-36de262b1c6f-kube-api-access-xbrz6\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.158537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.158438 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd88a946-654f-4e5d-a400-18d4b3a3ac3b-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:06.723815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.723744 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" Jul 06 20:21:06.723815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.723781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk" event={"ID":"8942ef1e-616a-4fc1-949c-36de262b1c6f","Type":"ContainerDied","Data":"bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a"} Jul 06 20:21:06.724222 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.723832 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb46831e34fe1070b0bd169919cd7f01ebcf16dd47538166124649d699fb64a" Jul 06 20:21:06.725937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.725874 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" Jul 06 20:21:06.725937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.725875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv" event={"ID":"dd88a946-654f-4e5d-a400-18d4b3a3ac3b","Type":"ContainerDied","Data":"a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7"} Jul 06 20:21:06.726105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.725954 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b654c428c8dbd032fe9eda372a30fc0a8afd45e0a0f97611ad01d5461f1be7" Jul 06 20:21:06.728305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.728285 2574 generic.go:356] "Generic (PLEG): container finished" podID="2d2ea925-982a-4417-a43a-255d64b2e864" containerID="862dcc0b3ae386491d77691465533a3c47075ce86043ca890d7781009c96139e" exitCode=0 Jul 06 20:21:06.728417 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.728366 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" event={"ID":"2d2ea925-982a-4417-a43a-255d64b2e864","Type":"ContainerDied","Data":"862dcc0b3ae386491d77691465533a3c47075ce86043ca890d7781009c96139e"} Jul 06 20:21:06.852056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.852034 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:06.963986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.963956 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util\") pod \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " Jul 06 20:21:06.964141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.964001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle\") pod \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " Jul 06 20:21:06.964141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.964060 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mdqg\" (UniqueName: \"kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg\") pod \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\" (UID: \"2b4d62be-a38d-48c4-813e-eb6de01bff2a\") " Jul 06 20:21:06.964493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.964462 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle" (OuterVolumeSpecName: "bundle") pod "2b4d62be-a38d-48c4-813e-eb6de01bff2a" (UID: "2b4d62be-a38d-48c4-813e-eb6de01bff2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:06.966151 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.966125 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg" (OuterVolumeSpecName: "kube-api-access-4mdqg") pod "2b4d62be-a38d-48c4-813e-eb6de01bff2a" (UID: "2b4d62be-a38d-48c4-813e-eb6de01bff2a"). InnerVolumeSpecName "kube-api-access-4mdqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:06.969260 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:06.969235 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util" (OuterVolumeSpecName: "util") pod "2b4d62be-a38d-48c4-813e-eb6de01bff2a" (UID: "2b4d62be-a38d-48c4-813e-eb6de01bff2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:07.064885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.064831 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mdqg\" (UniqueName: \"kubernetes.io/projected/2b4d62be-a38d-48c4-813e-eb6de01bff2a-kube-api-access-4mdqg\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:07.064885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.064853 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:07.064885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.064863 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b4d62be-a38d-48c4-813e-eb6de01bff2a-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:07.734714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.734687 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" Jul 06 20:21:07.735160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.734718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx" event={"ID":"2b4d62be-a38d-48c4-813e-eb6de01bff2a","Type":"ContainerDied","Data":"ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc"} Jul 06 20:21:07.735160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.734766 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef27c21854601d18f5dd100ec0488dac20fb2d0e2a3a49ac7b97e444aa968ebc" Jul 06 20:21:07.751369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.751344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:07.861263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.861242 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:07.972343 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.972317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg7lv\" (UniqueName: \"kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv\") pod \"2d2ea925-982a-4417-a43a-255d64b2e864\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " Jul 06 20:21:07.972483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.972382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle\") pod \"2d2ea925-982a-4417-a43a-255d64b2e864\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " Jul 06 20:21:07.972483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.972431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util\") pod \"2d2ea925-982a-4417-a43a-255d64b2e864\" (UID: \"2d2ea925-982a-4417-a43a-255d64b2e864\") " Jul 06 20:21:07.972859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.972831 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle" (OuterVolumeSpecName: "bundle") pod "2d2ea925-982a-4417-a43a-255d64b2e864" (UID: "2d2ea925-982a-4417-a43a-255d64b2e864"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:07.974373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.974356 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv" (OuterVolumeSpecName: "kube-api-access-jg7lv") pod "2d2ea925-982a-4417-a43a-255d64b2e864" (UID: "2d2ea925-982a-4417-a43a-255d64b2e864"). InnerVolumeSpecName "kube-api-access-jg7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:07.978270 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:07.978246 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util" (OuterVolumeSpecName: "util") pod "2d2ea925-982a-4417-a43a-255d64b2e864" (UID: "2d2ea925-982a-4417-a43a-255d64b2e864"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:08.073715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.073665 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:08.073715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.073683 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2ea925-982a-4417-a43a-255d64b2e864-util\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:08.073715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.073693 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7lv\" (UniqueName: \"kubernetes.io/projected/2d2ea925-982a-4417-a43a-255d64b2e864-kube-api-access-jg7lv\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:08.740822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.740785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" event={"ID":"2d2ea925-982a-4417-a43a-255d64b2e864","Type":"ContainerDied","Data":"02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570"} Jul 06 20:21:08.740822 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.740818 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd" Jul 06 20:21:08.741232 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:08.740860 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02f98305ec5240cae52ac804d076eb66ec4a3168fbe9d7e4655138ea46eb5570" Jul 06 20:21:12.719175 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719143 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719431 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719443 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719451 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="extract" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719456 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="extract" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719467 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719473 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719479 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719484 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="util" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719489 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="extract" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719494 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="extract" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719504 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="pull" Jul 06 20:21:12.719503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719509 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719514 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719520 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719527 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="util" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719532 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="util" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719538 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719543 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719550 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719555 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719562 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719566 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719576 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="pull" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719643 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b4d62be-a38d-48c4-813e-eb6de01bff2a" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719651 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd88a946-654f-4e5d-a400-18d4b3a3ac3b" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719657 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8942ef1e-616a-4fc1-949c-36de262b1c6f" containerName="extract" Jul 06 20:21:12.719971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.719662 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2ea925-982a-4417-a43a-255d64b2e864" containerName="extract" Jul 06 20:21:12.722730 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.722711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.732615 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.732589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:21:12.773852 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.773828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:12.814941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.814918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvc2\" (UniqueName: \"kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.814965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.814991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.815017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.815050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.815084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.815167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.815148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.875753 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.875732 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:12.916460 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvc2\" (UniqueName: \"kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.916741 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.916674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.917209 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.917185 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.917371 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.917348 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.917448 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.917352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.917448 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.917388 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.919004 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.918980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.919086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.919067 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:12.924491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:12.924474 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvc2\" (UniqueName: \"kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2\") pod \"console-6859dcd84b-wzf49\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:13.031749 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:13.031681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:13.158231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:13.158208 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:21:13.160304 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:13.160275 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f10f493_9c24_4c41_9a90_4e897969208e.slice/crio-7502fb123098686338fb7b21227895dce01b506e0bed68d08567abe39a8c6657 WatchSource:0}: Error finding container 7502fb123098686338fb7b21227895dce01b506e0bed68d08567abe39a8c6657: Status 404 returned error can't find the container with id 7502fb123098686338fb7b21227895dce01b506e0bed68d08567abe39a8c6657 Jul 06 20:21:13.768031 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:13.767992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6859dcd84b-wzf49" event={"ID":"9f10f493-9c24-4c41-9a90-4e897969208e","Type":"ContainerStarted","Data":"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0"} Jul 06 20:21:13.768463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:13.768084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6859dcd84b-wzf49" event={"ID":"9f10f493-9c24-4c41-9a90-4e897969208e","Type":"ContainerStarted","Data":"7502fb123098686338fb7b21227895dce01b506e0bed68d08567abe39a8c6657"} Jul 06 20:21:13.796305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:13.796247 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6859dcd84b-wzf49" podStartSLOduration=1.796231765 podStartE2EDuration="1.796231765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:13.794252366 +0000 UTC m=+390.284875916" watchObservedRunningTime="2026-07-06 20:21:13.796231765 +0000 UTC m=+390.286855315" Jul 06 20:21:14.326157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:14.326086 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:18.580980 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:18.580943 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:19.527173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.527138 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p"] Jul 06 20:21:19.530437 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.530422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.532739 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.532718 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kcwqt\"" Jul 06 20:21:19.548304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.548284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p"] Jul 06 20:21:19.571993 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.571970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8q9\" (UniqueName: \"kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.572094 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.572011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.673261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.673235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8q9\" (UniqueName: \"kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.673603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.673280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.673603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.673575 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.686082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.686056 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8q9\" (UniqueName: \"kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.725155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.725132 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:19.840534 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.840481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:19.981430 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:19.981407 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p"] Jul 06 20:21:19.983638 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:19.983607 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda951065_b484_4ed1_b2fc_18c2c5a24bac.slice/crio-81b9e0fca4b48e5985b66ab452e40f6982cdd091082b02dfc67a839151e6eee5 WatchSource:0}: Error finding container 81b9e0fca4b48e5985b66ab452e40f6982cdd091082b02dfc67a839151e6eee5: Status 404 returned error can't find the container with id 81b9e0fca4b48e5985b66ab452e40f6982cdd091082b02dfc67a839151e6eee5 Jul 06 20:21:20.129977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:20.129888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:20.808894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:20.808860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" event={"ID":"da951065-b484-4ed1-b2fc-18c2c5a24bac","Type":"ContainerStarted","Data":"81b9e0fca4b48e5985b66ab452e40f6982cdd091082b02dfc67a839151e6eee5"} Jul 06 20:21:21.153574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:21.153493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:21.267838 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:21.267804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:22.139447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.139368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lk47z"] Jul 06 20:21:22.143289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.143256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:22.146332 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.146314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ln4tl\"" Jul 06 20:21:22.161634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.161611 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lk47z"] Jul 06 20:21:22.192017 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.191996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9cf\" (UniqueName: \"kubernetes.io/projected/5512e65e-c821-4757-aae1-bfe75e050e71-kube-api-access-cl9cf\") pod \"authorino-operator-75cc8d5cb7-lk47z\" (UID: \"5512e65e-c821-4757-aae1-bfe75e050e71\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:22.292966 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.292940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9cf\" (UniqueName: \"kubernetes.io/projected/5512e65e-c821-4757-aae1-bfe75e050e71-kube-api-access-cl9cf\") pod \"authorino-operator-75cc8d5cb7-lk47z\" (UID: \"5512e65e-c821-4757-aae1-bfe75e050e71\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:22.305339 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.305321 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9cf\" (UniqueName: \"kubernetes.io/projected/5512e65e-c821-4757-aae1-bfe75e050e71-kube-api-access-cl9cf\") pod \"authorino-operator-75cc8d5cb7-lk47z\" (UID: \"5512e65e-c821-4757-aae1-bfe75e050e71\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:22.456894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.456730 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:22.673290 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:22.673263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5512e65e_c821_4757_aae1_bfe75e050e71.slice/crio-cc33653b2f174a541404ea48812ef570431b2f32d6c11d977cba4a5cb6bfa666 WatchSource:0}: Error finding container cc33653b2f174a541404ea48812ef570431b2f32d6c11d977cba4a5cb6bfa666: Status 404 returned error can't find the container with id cc33653b2f174a541404ea48812ef570431b2f32d6c11d977cba4a5cb6bfa666 Jul 06 20:21:22.681843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.681816 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-lk47z"] Jul 06 20:21:22.819963 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.819885 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" event={"ID":"5512e65e-c821-4757-aae1-bfe75e050e71","Type":"ContainerStarted","Data":"cc33653b2f174a541404ea48812ef570431b2f32d6c11d977cba4a5cb6bfa666"} Jul 06 20:21:22.828113 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.828083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:22.944756 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:22.944725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:23.031985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:23.031956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:23.032127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:23.032017 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:23.037008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:23.036989 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:23.833480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:23.833409 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:21:23.901840 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:23.901536 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:21:24.076680 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:24.076639 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:24.220592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:24.218119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:24.985920 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:24.985879 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:26.845593 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.845561 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:26.847765 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.847730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" event={"ID":"5512e65e-c821-4757-aae1-bfe75e050e71","Type":"ContainerStarted","Data":"41a3b0101d0a339da71251cb8fc3f60cd07097aafedba6f0bade2d9860c56b42"} Jul 06 20:21:26.847920 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.847877 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:26.849305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.849286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" event={"ID":"da951065-b484-4ed1-b2fc-18c2c5a24bac","Type":"ContainerStarted","Data":"6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a"} Jul 06 20:21:26.849415 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.849403 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:26.870449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.870402 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" podStartSLOduration=1.229625371 podStartE2EDuration="4.870383441s" totalImagesPullingTime="3.64075807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:26.86997981 +0000 UTC m=+403.360603360" watchObservedRunningTime="2026-07-06 20:21:26.870383441 +0000 UTC m=+403.361007002" Jul 06 20:21:26.902316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:26.902276 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" podStartSLOduration=1.567727697 podStartE2EDuration="7.902261893s" totalImagesPullingTime="6.334534196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:26.899975254 +0000 UTC m=+403.390598805" watchObservedRunningTime="2026-07-06 20:21:26.902261893 +0000 UTC m=+403.392885485" Jul 06 20:21:28.603979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:28.603944 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:28.736707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:28.736671 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:29.293707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.293671 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jp898"] Jul 06 20:21:29.297225 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.297205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.300096 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.300071 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 20:21:29.300191 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.300099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c29nn\"" Jul 06 20:21:29.300939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.300919 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 20:21:29.313196 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.313174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jp898"] Jul 06 20:21:29.354470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.354443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl965\" (UniqueName: \"kubernetes.io/projected/c917b062-7153-4753-901e-9ec634d1e77f-kube-api-access-dl965\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.354576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.354496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.354623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.354589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c917b062-7153-4753-901e-9ec634d1e77f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.355315 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.355295 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:29.455344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.455321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl965\" (UniqueName: \"kubernetes.io/projected/c917b062-7153-4753-901e-9ec634d1e77f-kube-api-access-dl965\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.455436 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.455355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.455436 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.455416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c917b062-7153-4753-901e-9ec634d1e77f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.455539 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:29.455523 2574 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 20:21:29.455595 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:29.455584 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert podName:c917b062-7153-4753-901e-9ec634d1e77f nodeName:}" failed. No retries permitted until 2026-07-06 20:21:29.955568008 +0000 UTC m=+406.446191537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jp898" (UID: "c917b062-7153-4753-901e-9ec634d1e77f") : secret "plugin-serving-cert" not found Jul 06 20:21:29.456052 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.456036 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c917b062-7153-4753-901e-9ec634d1e77f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.466235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.466217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl965\" (UniqueName: \"kubernetes.io/projected/c917b062-7153-4753-901e-9ec634d1e77f-kube-api-access-dl965\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.584819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.584756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:29.959251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.959221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:29.961564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:29.961543 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c917b062-7153-4753-901e-9ec634d1e77f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jp898\" (UID: \"c917b062-7153-4753-901e-9ec634d1e77f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:30.206492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:30.206461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" Jul 06 20:21:30.344492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:30.344465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jp898"] Jul 06 20:21:30.346630 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:30.346596 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc917b062_7153_4753_901e_9ec634d1e77f.slice/crio-5ed8444e374dc299f462c1c944ab6b783eb4f70620f385c212867bfa42c0b0a6 WatchSource:0}: Error finding container 5ed8444e374dc299f462c1c944ab6b783eb4f70620f385c212867bfa42c0b0a6: Status 404 returned error can't find the container with id 5ed8444e374dc299f462c1c944ab6b783eb4f70620f385c212867bfa42c0b0a6 Jul 06 20:21:30.870084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:30.870057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" event={"ID":"c917b062-7153-4753-901e-9ec634d1e77f","Type":"ContainerStarted","Data":"5ed8444e374dc299f462c1c944ab6b783eb4f70620f385c212867bfa42c0b0a6"} Jul 06 20:21:37.182661 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:37.182623 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:37.856631 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:37.856599 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-lk47z" Jul 06 20:21:37.856850 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:37.856827 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:38.655104 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:38.655073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:39.811493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:39.811461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:39.969069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:39.969034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:40.039768 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.039734 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p"] Jul 06 20:21:40.040008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.039971 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" containerName="manager" containerID="cri-o://6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a" gracePeriod=2 Jul 06 20:21:40.042559 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:40.042476 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" is forbidden: User \"system:node:ip-10-0-130-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-104.ec2.internal' and this object" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:40.044290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.044238 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p"] Jul 06 20:21:40.103776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.103697 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:21:40.104223 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.104206 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" containerName="manager" Jul 06 20:21:40.104302 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.104226 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" containerName="manager" Jul 06 20:21:40.104354 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.104327 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" containerName="manager" Jul 06 20:21:40.113281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.113259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.139528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.139501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:21:40.250571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.250540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlj4d\" (UniqueName: \"kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.250768 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.250629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.351700 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.351668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.351866 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.351781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlj4d\" (UniqueName: \"kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.352141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.352117 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.376122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.376044 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlj4d\" (UniqueName: \"kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lqxkc\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:40.424462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:40.424395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:41.217729 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.217684 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:21:41.224915 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.224877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.238117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.238095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:21:41.361532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.361498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmx2s\" (UniqueName: \"kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.361716 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.361547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.463061 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.463028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmx2s\" (UniqueName: \"kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.463243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.463071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.463446 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.463424 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.488829 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.488770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmx2s\" (UniqueName: \"kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s\") pod \"kuadrant-operator-controller-manager-86878c769f-4bnjb\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:41.538384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:41.538358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:44.398140 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:44.398055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:44.499634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:44.499597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:48.165335 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:48.165299 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:48.262840 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:48.262807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:48.931707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:48.931670 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-56f4b5f784-wh7kq" podUID="980bcd1d-5a1c-4a78-a205-79894d60a24a" containerName="console" containerID="cri-o://75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e" gracePeriod=15 Jul 06 20:21:54.356055 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.356027 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:21:54.366335 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.366312 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:54.387681 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:54.387654 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" is forbidden: User \"system:node:ip-10-0-130-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-104.ec2.internal' and this object" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:54.464691 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.464667 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:21:54.465687 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:54.465664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a230dc_50a8_4876_805a_24943449ab4b.slice/crio-fb3ca704d677b2070ff0e9efb38bd75902d260338e7aa1b2ab159aea112f187f WatchSource:0}: Error finding container fb3ca704d677b2070ff0e9efb38bd75902d260338e7aa1b2ab159aea112f187f: Status 404 returned error can't find the container with id fb3ca704d677b2070ff0e9efb38bd75902d260338e7aa1b2ab159aea112f187f Jul 06 20:21:54.470551 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.470529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:21:54.470983 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:21:54.470962 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c1d814_92d6_424c_8471_c7942a793415.slice/crio-f15d902c76baebcb02c77b88121ade99c2acbae8907fde40a9ac6dd1eb6afc3f WatchSource:0}: Error finding container f15d902c76baebcb02c77b88121ade99c2acbae8907fde40a9ac6dd1eb6afc3f: Status 404 returned error can't find the container with id f15d902c76baebcb02c77b88121ade99c2acbae8907fde40a9ac6dd1eb6afc3f Jul 06 20:21:54.477047 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz8q9\" (UniqueName: \"kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9\") pod \"da951065-b484-4ed1-b2fc-18c2c5a24bac\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " Jul 06 20:21:54.477128 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477062 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477128 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477200 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lxfg\" (UniqueName: \"kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477274 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477357 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume\") pod \"da951065-b484-4ed1-b2fc-18c2c5a24bac\" (UID: \"da951065-b484-4ed1-b2fc-18c2c5a24bac\") " Jul 06 20:21:54.477482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477370 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config" (OuterVolumeSpecName: "console-config") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:21:54.477482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477400 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477409 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca" (OuterVolumeSpecName: "service-ca") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:21:54.477482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config\") pod \"980bcd1d-5a1c-4a78-a205-79894d60a24a\" (UID: \"980bcd1d-5a1c-4a78-a205-79894d60a24a\") " Jul 06 20:21:54.477694 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477636 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:21:54.477694 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477661 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.477694 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477679 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.478006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.477964 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da951065-b484-4ed1-b2fc-18c2c5a24bac" (UID: "da951065-b484-4ed1-b2fc-18c2c5a24bac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:21:54.478093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.478032 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:21:54.479362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.479335 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:21:54.479362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.479338 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg" (OuterVolumeSpecName: "kube-api-access-5lxfg") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "kube-api-access-5lxfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:54.479501 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.479432 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9" (OuterVolumeSpecName: "kube-api-access-wz8q9") pod "da951065-b484-4ed1-b2fc-18c2c5a24bac" (UID: "da951065-b484-4ed1-b2fc-18c2c5a24bac"). InnerVolumeSpecName "kube-api-access-wz8q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:21:54.479501 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.479451 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "980bcd1d-5a1c-4a78-a205-79894d60a24a" (UID: "980bcd1d-5a1c-4a78-a205-79894d60a24a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:21:54.578961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.578888 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxfg\" (UniqueName: \"kubernetes.io/projected/980bcd1d-5a1c-4a78-a205-79894d60a24a-kube-api-access-5lxfg\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.578961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.578943 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.578961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.578959 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.579099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.578975 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da951065-b484-4ed1-b2fc-18c2c5a24bac-extensions-socket-volume\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.579099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.578989 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/980bcd1d-5a1c-4a78-a205-79894d60a24a-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.579099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.579003 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980bcd1d-5a1c-4a78-a205-79894d60a24a-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:54.579099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:54.579017 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8q9\" (UniqueName: \"kubernetes.io/projected/da951065-b484-4ed1-b2fc-18c2c5a24bac-kube-api-access-wz8q9\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:21:55.024886 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.024852 2574 generic.go:356] "Generic (PLEG): container finished" podID="980bcd1d-5a1c-4a78-a205-79894d60a24a" containerID="75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e" exitCode=0 Jul 06 20:21:55.025077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.024923 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56f4b5f784-wh7kq" Jul 06 20:21:55.025077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.024936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f4b5f784-wh7kq" event={"ID":"980bcd1d-5a1c-4a78-a205-79894d60a24a","Type":"ContainerDied","Data":"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e"} Jul 06 20:21:55.025077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.025009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56f4b5f784-wh7kq" event={"ID":"980bcd1d-5a1c-4a78-a205-79894d60a24a","Type":"ContainerDied","Data":"c12f07841f3a2730460d51a4852fcf31e51c17b0ea52cd5c7f66850cf7883875"} Jul 06 20:21:55.025077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.025033 2574 scope.go:117] "RemoveContainer" containerID="75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e" Jul 06 20:21:55.026846 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.026821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" event={"ID":"99c1d814-92d6-424c-8471-c7942a793415","Type":"ContainerStarted","Data":"d04914cf9f47d196d9317a1fff113de27e519e8614f8c1e97e5bb2ea0f482db9"} Jul 06 20:21:55.027006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.026869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" event={"ID":"99c1d814-92d6-424c-8471-c7942a793415","Type":"ContainerStarted","Data":"f15d902c76baebcb02c77b88121ade99c2acbae8907fde40a9ac6dd1eb6afc3f"} Jul 06 20:21:55.027006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.026958 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:21:55.028393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.028359 2574 generic.go:356] "Generic (PLEG): container finished" podID="da951065-b484-4ed1-b2fc-18c2c5a24bac" containerID="6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a" exitCode=0 Jul 06 20:21:55.028504 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.028409 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:55.030611 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.030430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" event={"ID":"12a230dc-50a8-4876-805a-24943449ab4b","Type":"ContainerStarted","Data":"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6"} Jul 06 20:21:55.030611 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.030538 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" event={"ID":"12a230dc-50a8-4876-805a-24943449ab4b","Type":"ContainerStarted","Data":"fb3ca704d677b2070ff0e9efb38bd75902d260338e7aa1b2ab159aea112f187f"} Jul 06 20:21:55.030611 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.030580 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:21:55.031567 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:55.031506 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" is forbidden: User \"system:node:ip-10-0-130-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-104.ec2.internal' and this object" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:55.032197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.032175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" event={"ID":"c917b062-7153-4753-901e-9ec634d1e77f","Type":"ContainerStarted","Data":"d7f074473b879ab73afd3dd098a942caaa2c220b58c27ed9f50cbba7b6a80701"} Jul 06 20:21:55.034935 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.034855 2574 scope.go:117] "RemoveContainer" containerID="75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e" Jul 06 20:21:55.035226 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:55.035207 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e\": container with ID starting with 75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e not found: ID does not exist" containerID="75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e" Jul 06 20:21:55.035319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.035232 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e"} err="failed to get container status \"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e\": rpc error: code = NotFound desc = could not find container \"75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e\": container with ID starting with 75bde33f258491c65530a4cef63203537448a6c290226aef3e95bfb4261e374e not found: ID does not exist" Jul 06 20:21:55.035319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.035248 2574 scope.go:117] "RemoveContainer" containerID="6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a" Jul 06 20:21:55.036657 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:55.036635 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" is forbidden: User \"system:node:ip-10-0-130-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-104.ec2.internal' and this object" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:55.043112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.043098 2574 scope.go:117] "RemoveContainer" containerID="6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a" Jul 06 20:21:55.043327 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:55.043308 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a\": container with ID starting with 6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a not found: ID does not exist" containerID="6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a" Jul 06 20:21:55.043387 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.043338 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a"} err="failed to get container status \"6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a\": rpc error: code = NotFound desc = could not find container \"6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a\": container with ID starting with 6c47bc6ae03e867a188ac4e5388d326a0092331a9b27436558038147f6ae810a not found: ID does not exist" Jul 06 20:21:55.086384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.086338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" podStartSLOduration=15.086326218 podStartE2EDuration="15.086326218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:55.083953409 +0000 UTC m=+431.574576965" watchObservedRunningTime="2026-07-06 20:21:55.086326218 +0000 UTC m=+431.576949768" Jul 06 20:21:55.115637 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.115599 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" podStartSLOduration=14.115587002 podStartE2EDuration="14.115587002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:55.115555061 +0000 UTC m=+431.606178611" watchObservedRunningTime="2026-07-06 20:21:55.115587002 +0000 UTC m=+431.606210552" Jul 06 20:21:55.138732 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.138707 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:21:55.147410 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.147385 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56f4b5f784-wh7kq"] Jul 06 20:21:55.170460 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:55.170416 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jp898" podStartSLOduration=2.141416553 podStartE2EDuration="26.170405658s" totalImagesPullingTime="24.028989105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:21:55.168556458 +0000 UTC m=+431.659180009" watchObservedRunningTime="2026-07-06 20:21:55.170405658 +0000 UTC m=+431.661029218" Jul 06 20:21:55.170637 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:21:55.170613 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dhr4p\" is forbidden: User \"system:node:ip-10-0-130-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-104.ec2.internal' and this object" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhr4p" Jul 06 20:21:56.071201 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:56.071172 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980bcd1d-5a1c-4a78-a205-79894d60a24a" path="/var/lib/kubelet/pods/980bcd1d-5a1c-4a78-a205-79894d60a24a/volumes" Jul 06 20:21:56.071561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:56.071491 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da951065-b484-4ed1-b2fc-18c2c5a24bac" path="/var/lib/kubelet/pods/da951065-b484-4ed1-b2fc-18c2c5a24bac/volumes" Jul 06 20:21:56.919154 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:56.919124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:57.020919 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:57.020886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:59.396481 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:59.396437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:21:59.495154 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:21:59.495125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:04.134223 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:04.134193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:04.234620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:04.234595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:06.041464 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.041389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:22:06.041464 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.041457 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:22:06.171157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.171127 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:22:06.171350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.171330 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" podUID="12a230dc-50a8-4876-805a-24943449ab4b" containerName="manager" containerID="cri-o://6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6" gracePeriod=10 Jul 06 20:22:06.412056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.412033 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:22:06.570045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.570017 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlj4d\" (UniqueName: \"kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d\") pod \"12a230dc-50a8-4876-805a-24943449ab4b\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " Jul 06 20:22:06.570170 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.570144 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume\") pod \"12a230dc-50a8-4876-805a-24943449ab4b\" (UID: \"12a230dc-50a8-4876-805a-24943449ab4b\") " Jul 06 20:22:06.570680 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.570653 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "12a230dc-50a8-4876-805a-24943449ab4b" (UID: "12a230dc-50a8-4876-805a-24943449ab4b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:22:06.572153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.572128 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d" (OuterVolumeSpecName: "kube-api-access-tlj4d") pod "12a230dc-50a8-4876-805a-24943449ab4b" (UID: "12a230dc-50a8-4876-805a-24943449ab4b"). InnerVolumeSpecName "kube-api-access-tlj4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:22:06.670894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.670871 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12a230dc-50a8-4876-805a-24943449ab4b-extensions-socket-volume\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:06.671008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:06.670895 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlj4d\" (UniqueName: \"kubernetes.io/projected/12a230dc-50a8-4876-805a-24943449ab4b-kube-api-access-tlj4d\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:07.099614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.099524 2574 generic.go:356] "Generic (PLEG): container finished" podID="12a230dc-50a8-4876-805a-24943449ab4b" containerID="6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6" exitCode=0 Jul 06 20:22:07.099614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.099568 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" Jul 06 20:22:07.099614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.099581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" event={"ID":"12a230dc-50a8-4876-805a-24943449ab4b","Type":"ContainerDied","Data":"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6"} Jul 06 20:22:07.100199 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.099619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc" event={"ID":"12a230dc-50a8-4876-805a-24943449ab4b","Type":"ContainerDied","Data":"fb3ca704d677b2070ff0e9efb38bd75902d260338e7aa1b2ab159aea112f187f"} Jul 06 20:22:07.100199 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.099642 2574 scope.go:117] "RemoveContainer" containerID="6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6" Jul 06 20:22:07.110232 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.110216 2574 scope.go:117] "RemoveContainer" containerID="6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6" Jul 06 20:22:07.110480 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:22:07.110460 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6\": container with ID starting with 6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6 not found: ID does not exist" containerID="6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6" Jul 06 20:22:07.110531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.110487 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6"} err="failed to get container status \"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6\": rpc error: code = NotFound desc = could not find container \"6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6\": container with ID starting with 6800862a395ee83de2d08f5cd9de4e9cd8391adc00f13ac76204ef451efa71f6 not found: ID does not exist" Jul 06 20:22:07.128423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.128401 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:22:07.131969 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:07.131947 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lqxkc"] Jul 06 20:22:08.071323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:08.071293 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a230dc-50a8-4876-805a-24943449ab4b" path="/var/lib/kubelet/pods/12a230dc-50a8-4876-805a-24943449ab4b/volumes" Jul 06 20:22:08.212215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:08.212185 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:08.335082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:08.335016 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:18.363184 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:18.363148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:18.468497 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:18.468466 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:18.629374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:18.629302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:19.368478 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:19.368445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:20.155950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:20.155895 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:20.252120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:20.252068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:22.671915 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.671870 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c"] Jul 06 20:22:22.672259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672215 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980bcd1d-5a1c-4a78-a205-79894d60a24a" containerName="console" Jul 06 20:22:22.672259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672227 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="980bcd1d-5a1c-4a78-a205-79894d60a24a" containerName="console" Jul 06 20:22:22.672259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672239 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a230dc-50a8-4876-805a-24943449ab4b" containerName="manager" Jul 06 20:22:22.672259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672245 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a230dc-50a8-4876-805a-24943449ab4b" containerName="manager" Jul 06 20:22:22.672410 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672311 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="980bcd1d-5a1c-4a78-a205-79894d60a24a" containerName="console" Jul 06 20:22:22.672410 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.672319 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a230dc-50a8-4876-805a-24943449ab4b" containerName="manager" Jul 06 20:22:22.701324 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.701293 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:22.708389 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.708361 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c"] Jul 06 20:22:22.708527 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.708471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.712110 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.712091 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-78lxd\"" Jul 06 20:22:22.786782 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.786877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfrm\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-kube-api-access-scfrm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.786877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.786877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94c2af7-b222-4c58-97b0-528de511c863-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.787010 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.787010 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.786943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.787078 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.787011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.787078 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.787041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.787141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.787076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94c2af7-b222-4c58-97b0-528de511c863-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.842153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.842129 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:22.887366 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94c2af7-b222-4c58-97b0-528de511c863-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94c2af7-b222-4c58-97b0-528de511c863-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfrm\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-kube-api-access-scfrm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887882 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887855 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.887977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887930 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.888026 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.887970 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.888099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.888080 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94c2af7-b222-4c58-97b0-528de511c863-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.888167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.888143 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.889666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.889650 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94c2af7-b222-4c58-97b0-528de511c863-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.890044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.890028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94c2af7-b222-4c58-97b0-528de511c863-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.895284 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.895264 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:22.895425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:22.895409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfrm\" (UniqueName: \"kubernetes.io/projected/d94c2af7-b222-4c58-97b0-528de511c863-kube-api-access-scfrm\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2qk7c\" (UID: \"d94c2af7-b222-4c58-97b0-528de511c863\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:23.018404 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:23.018348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:23.175199 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:23.175170 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c"] Jul 06 20:22:23.179071 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:23.179036 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94c2af7_b222_4c58_97b0_528de511c863.slice/crio-06aa77820fe98ac907608f48dc7635c23dd2380f36bdc8b1a8e1b14649191a23 WatchSource:0}: Error finding container 06aa77820fe98ac907608f48dc7635c23dd2380f36bdc8b1a8e1b14649191a23: Status 404 returned error can't find the container with id 06aa77820fe98ac907608f48dc7635c23dd2380f36bdc8b1a8e1b14649191a23 Jul 06 20:22:23.181577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:23.181541 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:22:23.181650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:23.181608 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:22:23.181650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:23.181635 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:22:24.186320 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:24.186279 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" event={"ID":"d94c2af7-b222-4c58-97b0-528de511c863","Type":"ContainerStarted","Data":"ea224f5da2d0bdd41211b60a1937d7ffcac0308b77ded8405fa0c03c7c626823"} Jul 06 20:22:24.186774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:24.186334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" event={"ID":"d94c2af7-b222-4c58-97b0-528de511c863","Type":"ContainerStarted","Data":"06aa77820fe98ac907608f48dc7635c23dd2380f36bdc8b1a8e1b14649191a23"} Jul 06 20:22:24.217574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:24.217531 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" podStartSLOduration=2.217513886 podStartE2EDuration="2.217513886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:24.215410128 +0000 UTC m=+460.706033678" watchObservedRunningTime="2026-07-06 20:22:24.217513886 +0000 UTC m=+460.708137438" Jul 06 20:22:24.590460 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:24.590380 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:24.762393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:24.762365 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:25.018797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.018766 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:25.023370 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.023349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:25.192618 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.192583 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:25.193460 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.193443 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:22:25.321291 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.321201 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:25.322259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.322237 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:25.361006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.360980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:25.361150 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.361086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.363511 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.363490 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 20:22:25.406095 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.406065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6848\" (UniqueName: \"kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.406278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.406211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.419634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.419609 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:25.423370 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.423327 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:25.507214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.507182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6848\" (UniqueName: \"kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.507375 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.507251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.507982 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.507959 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.518110 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.518086 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6848\" (UniqueName: \"kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848\") pod \"limitador-limitador-64c94fd64f-6xf9t\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.528674 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.528649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:25.673102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.673075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:25.847106 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:25.847080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:25.849109 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:25.849076 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded501db4_edde_46a7_84d5_1b6b94f3e1a0.slice/crio-a4c73bdd9918bfb5385353c2fcfbced9e5b9a20c3f94f710fb27d6a39d1cd5ec WatchSource:0}: Error finding container a4c73bdd9918bfb5385353c2fcfbced9e5b9a20c3f94f710fb27d6a39d1cd5ec: Status 404 returned error can't find the container with id a4c73bdd9918bfb5385353c2fcfbced9e5b9a20c3f94f710fb27d6a39d1cd5ec Jul 06 20:22:26.153330 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.153219 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:26.199943 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.199915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:26.200415 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.200012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:26.200869 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.200838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" event={"ID":"ed501db4-edde-46a7-84d5-1b6b94f3e1a0","Type":"ContainerStarted","Data":"a4c73bdd9918bfb5385353c2fcfbced9e5b9a20c3f94f710fb27d6a39d1cd5ec"} Jul 06 20:22:26.202967 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.202948 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2m962\"" Jul 06 20:22:26.208190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.207970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:26.214134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.214110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtq4\" (UniqueName: \"kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4\") pod \"authorino-58cd955675-tjrmb\" (UID: \"94ecb8b4-dd89-4d6e-820d-56855d27cec2\") " pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:26.311968 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.311937 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:26.315085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.315063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtq4\" (UniqueName: \"kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4\") pod \"authorino-58cd955675-tjrmb\" (UID: \"94ecb8b4-dd89-4d6e-820d-56855d27cec2\") " pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:26.324163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.324136 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtq4\" (UniqueName: \"kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4\") pod \"authorino-58cd955675-tjrmb\" (UID: \"94ecb8b4-dd89-4d6e-820d-56855d27cec2\") " pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:26.395710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.395689 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:26.416254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.416233 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:26.416347 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.416307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:26.510214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.510169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:26.517233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.517207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs2lh\" (UniqueName: \"kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh\") pod \"authorino-75d556bd77-vljxc\" (UID: \"07ed05f1-6d02-4cbf-b79f-f0e021740c85\") " pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:26.619004 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.618540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs2lh\" (UniqueName: \"kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh\") pod \"authorino-75d556bd77-vljxc\" (UID: \"07ed05f1-6d02-4cbf-b79f-f0e021740c85\") " pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:26.635413 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.635360 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs2lh\" (UniqueName: \"kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh\") pod \"authorino-75d556bd77-vljxc\" (UID: \"07ed05f1-6d02-4cbf-b79f-f0e021740c85\") " pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:26.677792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.677649 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:26.683366 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:26.683248 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ecb8b4_dd89_4d6e_820d_56855d27cec2.slice/crio-52b250d86cdb82089a86247fd0d4aa689e83b94053d321f68534d27ed2c4243c WatchSource:0}: Error finding container 52b250d86cdb82089a86247fd0d4aa689e83b94053d321f68534d27ed2c4243c: Status 404 returned error can't find the container with id 52b250d86cdb82089a86247fd0d4aa689e83b94053d321f68534d27ed2c4243c Jul 06 20:22:26.732587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.732559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:26.908867 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:26.908766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:26.910142 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:26.910112 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07ed05f1_6d02_4cbf_b79f_f0e021740c85.slice/crio-0632fa29a27b587939d6fde0f253d1dd10224488af249bf1b267649c1b7864b5 WatchSource:0}: Error finding container 0632fa29a27b587939d6fde0f253d1dd10224488af249bf1b267649c1b7864b5: Status 404 returned error can't find the container with id 0632fa29a27b587939d6fde0f253d1dd10224488af249bf1b267649c1b7864b5 Jul 06 20:22:27.207887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:27.207848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vljxc" event={"ID":"07ed05f1-6d02-4cbf-b79f-f0e021740c85","Type":"ContainerStarted","Data":"0632fa29a27b587939d6fde0f253d1dd10224488af249bf1b267649c1b7864b5"} Jul 06 20:22:27.208956 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:27.208932 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tjrmb" event={"ID":"94ecb8b4-dd89-4d6e-820d-56855d27cec2","Type":"ContainerStarted","Data":"52b250d86cdb82089a86247fd0d4aa689e83b94053d321f68534d27ed2c4243c"} Jul 06 20:22:31.233990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.233958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vljxc" event={"ID":"07ed05f1-6d02-4cbf-b79f-f0e021740c85","Type":"ContainerStarted","Data":"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d"} Jul 06 20:22:31.235420 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.235395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" event={"ID":"ed501db4-edde-46a7-84d5-1b6b94f3e1a0","Type":"ContainerStarted","Data":"67da737ca7fe33cc94479498739a33b885b7fcafe379f2cdde20ef4558e8bcfd"} Jul 06 20:22:31.235545 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.235467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:31.236785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.236762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tjrmb" event={"ID":"94ecb8b4-dd89-4d6e-820d-56855d27cec2","Type":"ContainerStarted","Data":"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1"} Jul 06 20:22:31.274368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.274328 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vljxc" podStartSLOduration=1.462761755 podStartE2EDuration="5.274315328s" totalImagesPullingTime="3.811553573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:31.271851835 +0000 UTC m=+467.762475387" watchObservedRunningTime="2026-07-06 20:22:31.274315328 +0000 UTC m=+467.764938878" Jul 06 20:22:31.322608 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.322559 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tjrmb" podStartSLOduration=1.296251573 podStartE2EDuration="5.322544221s" totalImagesPullingTime="4.026292648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:31.320550209 +0000 UTC m=+467.811173761" watchObservedRunningTime="2026-07-06 20:22:31.322544221 +0000 UTC m=+467.813167784" Jul 06 20:22:31.356973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.356926 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" podStartSLOduration=1.56345205 podStartE2EDuration="6.356891401s" totalImagesPullingTime="4.793439351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:31.356262806 +0000 UTC m=+467.846886357" watchObservedRunningTime="2026-07-06 20:22:31.356891401 +0000 UTC m=+467.847514953" Jul 06 20:22:31.382957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:31.382930 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:33.247715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:33.247675 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tjrmb" podUID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" containerName="authorino" containerID="cri-o://590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1" gracePeriod=30 Jul 06 20:22:33.491358 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:33.491336 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:33.595038 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:33.594980 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mtq4\" (UniqueName: \"kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4\") pod \"94ecb8b4-dd89-4d6e-820d-56855d27cec2\" (UID: \"94ecb8b4-dd89-4d6e-820d-56855d27cec2\") " Jul 06 20:22:33.597030 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:33.597007 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4" (OuterVolumeSpecName: "kube-api-access-7mtq4") pod "94ecb8b4-dd89-4d6e-820d-56855d27cec2" (UID: "94ecb8b4-dd89-4d6e-820d-56855d27cec2"). InnerVolumeSpecName "kube-api-access-7mtq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:22:33.696489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:33.696464 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtq4\" (UniqueName: \"kubernetes.io/projected/94ecb8b4-dd89-4d6e-820d-56855d27cec2-kube-api-access-7mtq4\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:34.254599 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.254563 2574 generic.go:356] "Generic (PLEG): container finished" podID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" containerID="590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1" exitCode=0 Jul 06 20:22:34.254936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.254604 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tjrmb" Jul 06 20:22:34.254936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.254606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tjrmb" event={"ID":"94ecb8b4-dd89-4d6e-820d-56855d27cec2","Type":"ContainerDied","Data":"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1"} Jul 06 20:22:34.254936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.254642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tjrmb" event={"ID":"94ecb8b4-dd89-4d6e-820d-56855d27cec2","Type":"ContainerDied","Data":"52b250d86cdb82089a86247fd0d4aa689e83b94053d321f68534d27ed2c4243c"} Jul 06 20:22:34.254936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.254681 2574 scope.go:117] "RemoveContainer" containerID="590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1" Jul 06 20:22:34.263039 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.263021 2574 scope.go:117] "RemoveContainer" containerID="590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1" Jul 06 20:22:34.263271 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:22:34.263251 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1\": container with ID starting with 590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1 not found: ID does not exist" containerID="590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1" Jul 06 20:22:34.263338 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.263283 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1"} err="failed to get container status \"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1\": rpc error: code = NotFound desc = could not find container \"590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1\": container with ID starting with 590c35d08605145bb359b00e35d5fd180e608532b2cb5b7a1908a8c5144687d1 not found: ID does not exist" Jul 06 20:22:34.281038 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.281016 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:34.286407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:34.286389 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tjrmb"] Jul 06 20:22:36.071643 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:36.071610 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" path="/var/lib/kubelet/pods/94ecb8b4-dd89-4d6e-820d-56855d27cec2/volumes" Jul 06 20:22:36.982868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:36.982840 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:37.049275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:37.049245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:39.867380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:39.867348 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:39.867797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:39.867529 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" podUID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" containerName="limitador" containerID="cri-o://67da737ca7fe33cc94479498739a33b885b7fcafe379f2cdde20ef4558e8bcfd" gracePeriod=30 Jul 06 20:22:39.868168 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:39.868069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:40.288429 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.288405 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" containerID="67da737ca7fe33cc94479498739a33b885b7fcafe379f2cdde20ef4558e8bcfd" exitCode=0 Jul 06 20:22:40.288537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.288476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" event={"ID":"ed501db4-edde-46a7-84d5-1b6b94f3e1a0","Type":"ContainerDied","Data":"67da737ca7fe33cc94479498739a33b885b7fcafe379f2cdde20ef4558e8bcfd"} Jul 06 20:22:40.415173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.415149 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:40.547983 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.547898 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file\") pod \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " Jul 06 20:22:40.547983 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.547956 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6848\" (UniqueName: \"kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848\") pod \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\" (UID: \"ed501db4-edde-46a7-84d5-1b6b94f3e1a0\") " Jul 06 20:22:40.548229 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.548205 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file" (OuterVolumeSpecName: "config-file") pod "ed501db4-edde-46a7-84d5-1b6b94f3e1a0" (UID: "ed501db4-edde-46a7-84d5-1b6b94f3e1a0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:22:40.549938 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.549916 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848" (OuterVolumeSpecName: "kube-api-access-l6848") pod "ed501db4-edde-46a7-84d5-1b6b94f3e1a0" (UID: "ed501db4-edde-46a7-84d5-1b6b94f3e1a0"). InnerVolumeSpecName "kube-api-access-l6848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:22:40.649052 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.649028 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-config-file\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:40.649052 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:40.649049 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6848\" (UniqueName: \"kubernetes.io/projected/ed501db4-edde-46a7-84d5-1b6b94f3e1a0-kube-api-access-l6848\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:41.295164 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.295134 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" Jul 06 20:22:41.295550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.295134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6xf9t" event={"ID":"ed501db4-edde-46a7-84d5-1b6b94f3e1a0","Type":"ContainerDied","Data":"a4c73bdd9918bfb5385353c2fcfbced9e5b9a20c3f94f710fb27d6a39d1cd5ec"} Jul 06 20:22:41.295550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.295267 2574 scope.go:117] "RemoveContainer" containerID="67da737ca7fe33cc94479498739a33b885b7fcafe379f2cdde20ef4558e8bcfd" Jul 06 20:22:41.317709 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.317688 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:41.321576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.321552 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6xf9t"] Jul 06 20:22:41.751643 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.751621 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-22d2g"] Jul 06 20:22:41.751990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.751978 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" containerName="limitador" Jul 06 20:22:41.752053 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.751991 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" containerName="limitador" Jul 06 20:22:41.752053 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.752001 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" containerName="authorino" Jul 06 20:22:41.752053 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.752006 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" containerName="authorino" Jul 06 20:22:41.752156 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.752056 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" containerName="limitador" Jul 06 20:22:41.752156 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.752068 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ecb8b4-dd89-4d6e-820d-56855d27cec2" containerName="authorino" Jul 06 20:22:41.755039 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.755016 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.757438 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.757418 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 20:22:41.757696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.757675 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dpmtz\"" Jul 06 20:22:41.766727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.766706 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-22d2g"] Jul 06 20:22:41.795317 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.795284 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:41.858766 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.858740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8hf\" (UniqueName: \"kubernetes.io/projected/f5397d8c-4e76-4503-be19-76022375845c-kube-api-access-zh8hf\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.858855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.858778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5397d8c-4e76-4503-be19-76022375845c-data\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.959217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.959182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8hf\" (UniqueName: \"kubernetes.io/projected/f5397d8c-4e76-4503-be19-76022375845c-kube-api-access-zh8hf\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.959357 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.959231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5397d8c-4e76-4503-be19-76022375845c-data\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.959581 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.959566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5397d8c-4e76-4503-be19-76022375845c-data\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:41.968850 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:41.968827 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8hf\" (UniqueName: \"kubernetes.io/projected/f5397d8c-4e76-4503-be19-76022375845c-kube-api-access-zh8hf\") pod \"postgres-69fb4594cf-22d2g\" (UID: \"f5397d8c-4e76-4503-be19-76022375845c\") " pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:42.066556 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:42.066496 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:42.072344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:42.072320 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed501db4-edde-46a7-84d5-1b6b94f3e1a0" path="/var/lib/kubelet/pods/ed501db4-edde-46a7-84d5-1b6b94f3e1a0/volumes" Jul 06 20:22:42.195885 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:42.195856 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-22d2g"] Jul 06 20:22:42.197394 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:42.197368 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5397d8c_4e76_4503_be19_76022375845c.slice/crio-23a75cbd7c3c5ae9618ab7815b7a5cbbd161d4ddc662158612adb5b3d15c0ef2 WatchSource:0}: Error finding container 23a75cbd7c3c5ae9618ab7815b7a5cbbd161d4ddc662158612adb5b3d15c0ef2: Status 404 returned error can't find the container with id 23a75cbd7c3c5ae9618ab7815b7a5cbbd161d4ddc662158612adb5b3d15c0ef2 Jul 06 20:22:42.301651 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:42.301613 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-22d2g" event={"ID":"f5397d8c-4e76-4503-be19-76022375845c","Type":"ContainerStarted","Data":"23a75cbd7c3c5ae9618ab7815b7a5cbbd161d4ddc662158612adb5b3d15c0ef2"} Jul 06 20:22:47.063951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:47.063924 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 20:22:47.337856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:47.337782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-22d2g" event={"ID":"f5397d8c-4e76-4503-be19-76022375845c","Type":"ContainerStarted","Data":"af11ef09c48b964f9cc9083015ba3b86a1a5975bd45c92c880415f9ddc4c39cf"} Jul 06 20:22:47.338007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:47.337938 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:47.355858 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:47.355807 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-22d2g" podStartSLOduration=1.493315881 podStartE2EDuration="6.355795178s" totalImagesPullingTime="4.862479297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:47.353531893 +0000 UTC m=+483.844155443" watchObservedRunningTime="2026-07-06 20:22:47.355795178 +0000 UTC m=+483.846418728" Jul 06 20:22:53.374038 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:53.374011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-22d2g" Jul 06 20:22:53.704282 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:53.704253 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:53.958548 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:53.958479 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:53.964528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:53.964510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:53.970191 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:53.970169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:54.056471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.056443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhm7\" (UniqueName: \"kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7\") pod \"authorino-5d4b67c446-9s2jz\" (UID: \"4fed38d5-b006-47e6-beb2-62e820fd6286\") " pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:54.146295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.146263 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:22:54.157440 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.157412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhm7\" (UniqueName: \"kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7\") pod \"authorino-5d4b67c446-9s2jz\" (UID: \"4fed38d5-b006-47e6-beb2-62e820fd6286\") " pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:54.166028 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.166005 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhm7\" (UniqueName: \"kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7\") pod \"authorino-5d4b67c446-9s2jz\" (UID: \"4fed38d5-b006-47e6-beb2-62e820fd6286\") " pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:54.223925 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.223849 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:54.224106 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.224094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:54.356307 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.356281 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:54.358334 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:54.358311 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fed38d5_b006_47e6_beb2_62e820fd6286.slice/crio-61ab1dc09145a8304506991e3e98d046cf372c31b9b1905ebbfce43180d56b6b WatchSource:0}: Error finding container 61ab1dc09145a8304506991e3e98d046cf372c31b9b1905ebbfce43180d56b6b: Status 404 returned error can't find the container with id 61ab1dc09145a8304506991e3e98d046cf372c31b9b1905ebbfce43180d56b6b Jul 06 20:22:54.378657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.378631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" event={"ID":"4fed38d5-b006-47e6-beb2-62e820fd6286","Type":"ContainerStarted","Data":"61ab1dc09145a8304506991e3e98d046cf372c31b9b1905ebbfce43180d56b6b"} Jul 06 20:22:54.607036 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.606980 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:22:54.612203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.612185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.614647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.614629 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 20:22:54.621657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.621631 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:22:54.762119 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.762095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.762234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.762174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dl6\" (UniqueName: \"kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.863194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.863136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.863289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.863209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dl6\" (UniqueName: \"kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.865447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.865422 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.870843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.870824 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dl6\" (UniqueName: \"kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6\") pod \"authorino-57884f4449-p568p\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:54.921504 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:54.921484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:22:55.055853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.055830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:22:55.057932 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:22:55.057883 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ffe854_3810_488f_b756_7a2bd64fb4ad.slice/crio-2d41ac52632cb6da6ad0461813588541c81748e2502e611644219f997db34caa WatchSource:0}: Error finding container 2d41ac52632cb6da6ad0461813588541c81748e2502e611644219f997db34caa: Status 404 returned error can't find the container with id 2d41ac52632cb6da6ad0461813588541c81748e2502e611644219f997db34caa Jul 06 20:22:55.385024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.384994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57884f4449-p568p" event={"ID":"72ffe854-3810-488f-b756-7a2bd64fb4ad","Type":"ContainerStarted","Data":"2d41ac52632cb6da6ad0461813588541c81748e2502e611644219f997db34caa"} Jul 06 20:22:55.386422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.386398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" event={"ID":"4fed38d5-b006-47e6-beb2-62e820fd6286","Type":"ContainerStarted","Data":"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3"} Jul 06 20:22:55.386533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.386492 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" podUID="4fed38d5-b006-47e6-beb2-62e820fd6286" containerName="authorino" containerID="cri-o://6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3" gracePeriod=30 Jul 06 20:22:55.402358 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.402319 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" podStartSLOduration=1.808941296 podStartE2EDuration="2.402305127s" totalImagesPullingTime="593.363831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:55.401393999 +0000 UTC m=+491.892017621" watchObservedRunningTime="2026-07-06 20:22:55.402305127 +0000 UTC m=+491.892929041" Jul 06 20:22:55.639449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.639392 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:55.770740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.770710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbhm7\" (UniqueName: \"kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7\") pod \"4fed38d5-b006-47e6-beb2-62e820fd6286\" (UID: \"4fed38d5-b006-47e6-beb2-62e820fd6286\") " Jul 06 20:22:55.772760 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.772729 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7" (OuterVolumeSpecName: "kube-api-access-kbhm7") pod "4fed38d5-b006-47e6-beb2-62e820fd6286" (UID: "4fed38d5-b006-47e6-beb2-62e820fd6286"). InnerVolumeSpecName "kube-api-access-kbhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:22:55.872066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:55.872041 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhm7\" (UniqueName: \"kubernetes.io/projected/4fed38d5-b006-47e6-beb2-62e820fd6286-kube-api-access-kbhm7\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:56.392104 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.392023 2574 generic.go:356] "Generic (PLEG): container finished" podID="4fed38d5-b006-47e6-beb2-62e820fd6286" containerID="6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3" exitCode=0 Jul 06 20:22:56.392104 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.392060 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" Jul 06 20:22:56.392555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.392109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" event={"ID":"4fed38d5-b006-47e6-beb2-62e820fd6286","Type":"ContainerDied","Data":"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3"} Jul 06 20:22:56.392555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.392148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-9s2jz" event={"ID":"4fed38d5-b006-47e6-beb2-62e820fd6286","Type":"ContainerDied","Data":"61ab1dc09145a8304506991e3e98d046cf372c31b9b1905ebbfce43180d56b6b"} Jul 06 20:22:56.392555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.392164 2574 scope.go:117] "RemoveContainer" containerID="6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3" Jul 06 20:22:56.393712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.393685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57884f4449-p568p" event={"ID":"72ffe854-3810-488f-b756-7a2bd64fb4ad","Type":"ContainerStarted","Data":"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3"} Jul 06 20:22:56.400830 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.400812 2574 scope.go:117] "RemoveContainer" containerID="6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3" Jul 06 20:22:56.401080 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:22:56.401049 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3\": container with ID starting with 6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3 not found: ID does not exist" containerID="6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3" Jul 06 20:22:56.401142 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.401090 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3"} err="failed to get container status \"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3\": rpc error: code = NotFound desc = could not find container \"6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3\": container with ID starting with 6118ce52d9a37bc16e4b426b6b1a87cb699bddfa1e5088f2879515a65c4042e3 not found: ID does not exist" Jul 06 20:22:56.413426 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.413384 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57884f4449-p568p" podStartSLOduration=2.0620952 podStartE2EDuration="2.413373794s" totalImagesPullingTime="351.278594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:56.410869864 +0000 UTC m=+492.901493415" watchObservedRunningTime="2026-07-06 20:22:56.413373794 +0000 UTC m=+492.903997343" Jul 06 20:22:56.429593 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.429567 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:56.437397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.437371 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9s2jz"] Jul 06 20:22:56.442095 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.442071 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:56.442236 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.442217 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vljxc" podUID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" containerName="authorino" containerID="cri-o://83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d" gracePeriod=30 Jul 06 20:22:56.691212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.691185 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:56.780108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.780084 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs2lh\" (UniqueName: \"kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh\") pod \"07ed05f1-6d02-4cbf-b79f-f0e021740c85\" (UID: \"07ed05f1-6d02-4cbf-b79f-f0e021740c85\") " Jul 06 20:22:56.781951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.781923 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh" (OuterVolumeSpecName: "kube-api-access-bs2lh") pod "07ed05f1-6d02-4cbf-b79f-f0e021740c85" (UID: "07ed05f1-6d02-4cbf-b79f-f0e021740c85"). InnerVolumeSpecName "kube-api-access-bs2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:22:56.880784 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:56.880755 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs2lh\" (UniqueName: \"kubernetes.io/projected/07ed05f1-6d02-4cbf-b79f-f0e021740c85-kube-api-access-bs2lh\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:22:57.401245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.401213 2574 generic.go:356] "Generic (PLEG): container finished" podID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" containerID="83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d" exitCode=0 Jul 06 20:22:57.401675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.401256 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vljxc" Jul 06 20:22:57.401675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.401277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vljxc" event={"ID":"07ed05f1-6d02-4cbf-b79f-f0e021740c85","Type":"ContainerDied","Data":"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d"} Jul 06 20:22:57.401675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.401326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vljxc" event={"ID":"07ed05f1-6d02-4cbf-b79f-f0e021740c85","Type":"ContainerDied","Data":"0632fa29a27b587939d6fde0f253d1dd10224488af249bf1b267649c1b7864b5"} Jul 06 20:22:57.401675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.401348 2574 scope.go:117] "RemoveContainer" containerID="83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d" Jul 06 20:22:57.410645 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.410630 2574 scope.go:117] "RemoveContainer" containerID="83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d" Jul 06 20:22:57.410892 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:22:57.410876 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d\": container with ID starting with 83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d not found: ID does not exist" containerID="83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d" Jul 06 20:22:57.411013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.410994 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d"} err="failed to get container status \"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d\": rpc error: code = NotFound desc = could not find container \"83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d\": container with ID starting with 83f674f6e58501e0ab67514578c0bdc8492fa8059ac3ab9233109c5f7c11fa8d not found: ID does not exist" Jul 06 20:22:57.422585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.422564 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:57.426244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:57.426222 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vljxc"] Jul 06 20:22:58.071872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:58.071842 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" path="/var/lib/kubelet/pods/07ed05f1-6d02-4cbf-b79f-f0e021740c85/volumes" Jul 06 20:22:58.072188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:22:58.072174 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fed38d5-b006-47e6-beb2-62e820fd6286" path="/var/lib/kubelet/pods/4fed38d5-b006-47e6-beb2-62e820fd6286/volumes" Jul 06 20:23:00.728261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.728233 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:00.799629 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.799585 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:00.799985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.799971 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" containerName="authorino" Jul 06 20:23:00.799985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.799986 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" containerName="authorino" Jul 06 20:23:00.800079 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.800004 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fed38d5-b006-47e6-beb2-62e820fd6286" containerName="authorino" Jul 06 20:23:00.800079 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.800010 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fed38d5-b006-47e6-beb2-62e820fd6286" containerName="authorino" Jul 06 20:23:00.800079 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.800069 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ed05f1-6d02-4cbf-b79f-f0e021740c85" containerName="authorino" Jul 06 20:23:00.800170 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.800082 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fed38d5-b006-47e6-beb2-62e820fd6286" containerName="authorino" Jul 06 20:23:00.913528 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.913498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:00.985817 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.985749 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:00.985817 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.985793 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:23:00.986011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.985863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:00.989279 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.989259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:23:00.989411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.989259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7vbpg\"" Jul 06 20:23:00.989411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.989294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:23:00.996839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:00.996814 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:01.013220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.013193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:23:01.013341 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.013298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.114127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.114091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.114127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.114126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnzd\" (UniqueName: \"kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.114343 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.114146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdngl\" (UniqueName: \"kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.114343 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.114254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.215499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.215466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.215499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.215500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnzd\" (UniqueName: \"kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.215712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.215520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdngl\" (UniqueName: \"kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.215712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.215594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.218134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.218112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.218243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.218144 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.224720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.224697 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdngl\" (UniqueName: \"kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl\") pod \"maas-controller-d79b4b87b-59tql\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.226941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.226916 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnzd\" (UniqueName: \"kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd\") pod \"maas-controller-7f66685579-czd94\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.297076 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.297019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:01.322769 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.322749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:01.459456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.459425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:01.459585 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:23:01.459427 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d76a2bb_2b1e_482c_b5f3_4d569004c33b.slice/crio-7d61548ef133686c0beddcb6cd28da586fe8c70d6495cdbcfdda54e8a2206d70 WatchSource:0}: Error finding container 7d61548ef133686c0beddcb6cd28da586fe8c70d6495cdbcfdda54e8a2206d70: Status 404 returned error can't find the container with id 7d61548ef133686c0beddcb6cd28da586fe8c70d6495cdbcfdda54e8a2206d70 Jul 06 20:23:01.483832 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:01.483808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:23:01.485154 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:23:01.485131 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac934f11_22e7_4f54_b7c3_a08e7a2e8231.slice/crio-573b37fd81ed240a98689de8ae72c044d7f44d6c3d9e3ce1d9a7ba00c42d1ee4 WatchSource:0}: Error finding container 573b37fd81ed240a98689de8ae72c044d7f44d6c3d9e3ce1d9a7ba00c42d1ee4: Status 404 returned error can't find the container with id 573b37fd81ed240a98689de8ae72c044d7f44d6c3d9e3ce1d9a7ba00c42d1ee4 Jul 06 20:23:02.434819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:02.434781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-59tql" event={"ID":"ac934f11-22e7-4f54-b7c3-a08e7a2e8231","Type":"ContainerStarted","Data":"573b37fd81ed240a98689de8ae72c044d7f44d6c3d9e3ce1d9a7ba00c42d1ee4"} Jul 06 20:23:02.436584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:02.436552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f66685579-czd94" event={"ID":"9d76a2bb-2b1e-482c-b5f3-4d569004c33b","Type":"ContainerStarted","Data":"7d61548ef133686c0beddcb6cd28da586fe8c70d6495cdbcfdda54e8a2206d70"} Jul 06 20:23:03.813412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:03.813312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:03.896127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:03.896099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:04.448348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.448314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-59tql" event={"ID":"ac934f11-22e7-4f54-b7c3-a08e7a2e8231","Type":"ContainerStarted","Data":"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e"} Jul 06 20:23:04.448535 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.448519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:04.449758 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.449732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f66685579-czd94" event={"ID":"9d76a2bb-2b1e-482c-b5f3-4d569004c33b","Type":"ContainerStarted","Data":"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31"} Jul 06 20:23:04.449898 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.449783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:04.467190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.467146 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d79b4b87b-59tql" podStartSLOduration=2.387543885 podStartE2EDuration="4.467133459s" totalImagesPullingTime="2.079589574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:23:04.465084282 +0000 UTC m=+500.955707832" watchObservedRunningTime="2026-07-06 20:23:04.467133459 +0000 UTC m=+500.957757009" Jul 06 20:23:04.482735 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:04.482696 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f66685579-czd94" podStartSLOduration=2.367498741 podStartE2EDuration="4.482682553s" totalImagesPullingTime="2.115183812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:23:04.481065114 +0000 UTC m=+500.971688665" watchObservedRunningTime="2026-07-06 20:23:04.482682553 +0000 UTC m=+500.973306139" Jul 06 20:23:15.460999 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.460970 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:23:15.461576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.461319 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:15.535675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.535645 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:15.535857 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.535821 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7f66685579-czd94" podUID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" containerName="manager" containerID="cri-o://4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31" gracePeriod=30 Jul 06 20:23:15.784621 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.784598 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:15.827268 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.827238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhnzd\" (UniqueName: \"kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd\") pod \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " Jul 06 20:23:15.827402 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.827291 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert\") pod \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\" (UID: \"9d76a2bb-2b1e-482c-b5f3-4d569004c33b\") " Jul 06 20:23:15.829307 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.829281 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd" (OuterVolumeSpecName: "kube-api-access-qhnzd") pod "9d76a2bb-2b1e-482c-b5f3-4d569004c33b" (UID: "9d76a2bb-2b1e-482c-b5f3-4d569004c33b"). InnerVolumeSpecName "kube-api-access-qhnzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:23:15.829307 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.829291 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9d76a2bb-2b1e-482c-b5f3-4d569004c33b" (UID: "9d76a2bb-2b1e-482c-b5f3-4d569004c33b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:23:15.928458 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.928437 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-webhook-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:23:15.928558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:15.928459 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhnzd\" (UniqueName: \"kubernetes.io/projected/9d76a2bb-2b1e-482c-b5f3-4d569004c33b-kube-api-access-qhnzd\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:23:16.510949 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.510897 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" containerID="4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31" exitCode=0 Jul 06 20:23:16.511297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.510954 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f66685579-czd94" Jul 06 20:23:16.511297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.510995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f66685579-czd94" event={"ID":"9d76a2bb-2b1e-482c-b5f3-4d569004c33b","Type":"ContainerDied","Data":"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31"} Jul 06 20:23:16.511297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.511042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f66685579-czd94" event={"ID":"9d76a2bb-2b1e-482c-b5f3-4d569004c33b","Type":"ContainerDied","Data":"7d61548ef133686c0beddcb6cd28da586fe8c70d6495cdbcfdda54e8a2206d70"} Jul 06 20:23:16.511297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.511067 2574 scope.go:117] "RemoveContainer" containerID="4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31" Jul 06 20:23:16.519452 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.519438 2574 scope.go:117] "RemoveContainer" containerID="4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31" Jul 06 20:23:16.519688 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:23:16.519668 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31\": container with ID starting with 4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31 not found: ID does not exist" containerID="4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31" Jul 06 20:23:16.519759 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.519697 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31"} err="failed to get container status \"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31\": rpc error: code = NotFound desc = could not find container \"4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31\": container with ID starting with 4bf72f46d472a6004c8892c34db33d85025a88ac8377c7ae03f4d8164369fb31 not found: ID does not exist" Jul 06 20:23:16.528632 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.528613 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:16.532436 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:16.532417 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f66685579-czd94"] Jul 06 20:23:18.071485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:18.071452 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" path="/var/lib/kubelet/pods/9d76a2bb-2b1e-482c-b5f3-4d569004c33b/volumes" Jul 06 20:23:36.575583 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:36.575482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:37.058628 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.058582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:37.107815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.107783 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-688c79454f-bqw54"] Jul 06 20:23:37.108163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.108147 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" containerName="manager" Jul 06 20:23:37.108239 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.108166 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" containerName="manager" Jul 06 20:23:37.108278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.108258 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d76a2bb-2b1e-482c-b5f3-4d569004c33b" containerName="manager" Jul 06 20:23:37.111655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.111636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.114683 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.114659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-5t798\"" Jul 06 20:23:37.115011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.114993 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 20:23:37.152572 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.152548 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-688c79454f-bqw54"] Jul 06 20:23:37.192469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.192436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zgv\" (UniqueName: \"kubernetes.io/projected/61ef62ac-1d3d-4028-a95c-4c219060a3c9-kube-api-access-87zgv\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.192620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.192475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ef62ac-1d3d-4028-a95c-4c219060a3c9-maas-api-tls\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.192620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.192586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ef62ac-1d3d-4028-a95c-4c219060a3c9-openshift-service-ca\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.212609 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.212580 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:37.293561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.293529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ef62ac-1d3d-4028-a95c-4c219060a3c9-openshift-service-ca\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.293705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.293606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zgv\" (UniqueName: \"kubernetes.io/projected/61ef62ac-1d3d-4028-a95c-4c219060a3c9-kube-api-access-87zgv\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.293705 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.293636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ef62ac-1d3d-4028-a95c-4c219060a3c9-maas-api-tls\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.294253 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.294222 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ef62ac-1d3d-4028-a95c-4c219060a3c9-openshift-service-ca\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.296201 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.296174 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ef62ac-1d3d-4028-a95c-4c219060a3c9-maas-api-tls\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.302795 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.302758 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zgv\" (UniqueName: \"kubernetes.io/projected/61ef62ac-1d3d-4028-a95c-4c219060a3c9-kube-api-access-87zgv\") pod \"maas-api-688c79454f-bqw54\" (UID: \"61ef62ac-1d3d-4028-a95c-4c219060a3c9\") " pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.421637 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.421604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:37.507215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.507167 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:37.572990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.572964 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-688c79454f-bqw54"] Jul 06 20:23:37.574774 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:23:37.574746 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ef62ac_1d3d_4028_a95c_4c219060a3c9.slice/crio-f1cff05160e644f8d5853749696e7dcdbe51738e75573cda8609412dbf2da3f1 WatchSource:0}: Error finding container f1cff05160e644f8d5853749696e7dcdbe51738e75573cda8609412dbf2da3f1: Status 404 returned error can't find the container with id f1cff05160e644f8d5853749696e7dcdbe51738e75573cda8609412dbf2da3f1 Jul 06 20:23:37.612898 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.612865 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:37.625167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:37.625143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-688c79454f-bqw54" event={"ID":"61ef62ac-1d3d-4028-a95c-4c219060a3c9","Type":"ContainerStarted","Data":"f1cff05160e644f8d5853749696e7dcdbe51738e75573cda8609412dbf2da3f1"} Jul 06 20:23:40.643527 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:40.643489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-688c79454f-bqw54" event={"ID":"61ef62ac-1d3d-4028-a95c-4c219060a3c9","Type":"ContainerStarted","Data":"c056b98e34d4edb9b20df9df247a1002731c2c9edf1796b0aa04f2284f82f497"} Jul 06 20:23:40.643966 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:40.643563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:40.661515 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:40.661464 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-688c79454f-bqw54" podStartSLOduration=1.529717544 podStartE2EDuration="3.661424015s" totalImagesPullingTime="2.131706471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:23:40.661048138 +0000 UTC m=+537.151671710" watchObservedRunningTime="2026-07-06 20:23:40.661424015 +0000 UTC m=+537.152047566" Jul 06 20:23:45.674893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.674863 2574 generic.go:356] "Generic (PLEG): container finished" podID="d94c2af7-b222-4c58-97b0-528de511c863" containerID="ea224f5da2d0bdd41211b60a1937d7ffcac0308b77ded8405fa0c03c7c626823" exitCode=0 Jul 06 20:23:45.675419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.674937 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" event={"ID":"d94c2af7-b222-4c58-97b0-528de511c863","Type":"ContainerDied","Data":"ea224f5da2d0bdd41211b60a1937d7ffcac0308b77ded8405fa0c03c7c626823"} Jul 06 20:23:45.675419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.675261 2574 scope.go:117] "RemoveContainer" containerID="ea224f5da2d0bdd41211b60a1937d7ffcac0308b77ded8405fa0c03c7c626823" Jul 06 20:23:45.676178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.676150 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:23:45.676279 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.676232 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:23:45.676342 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:45.676279 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:23:46.655228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:46.655194 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-688c79454f-bqw54" Jul 06 20:23:46.690311 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:46.689400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" event={"ID":"d94c2af7-b222-4c58-97b0-528de511c863","Type":"ContainerStarted","Data":"6650c15b7a893426ee0e16308a0842828bd39f6ad1f116aad5b3f033ab3f2875"} Jul 06 20:23:47.019138 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:47.019042 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:23:47.019316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:47.019259 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2qk7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 20:23:47.019393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:47.019311 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" podUID="d94c2af7-b222-4c58-97b0-528de511c863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 20:23:47.627922 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:47.627862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:47.727003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:47.726970 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:23:48.019396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:48.019361 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2qk7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 20:23:48.019593 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:48.019432 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" podUID="d94c2af7-b222-4c58-97b0-528de511c863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 20:23:49.019485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:49.019441 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2qk7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 20:23:49.019855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:49.019504 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" podUID="d94c2af7-b222-4c58-97b0-528de511c863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 20:23:50.019632 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:50.019598 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2qk7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 06 20:23:50.020052 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:50.019655 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" podUID="d94c2af7-b222-4c58-97b0-528de511c863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 06 20:23:51.022221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:51.022192 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:23:51.022727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:51.022707 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:23:51.023385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:23:51.023369 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2qk7c" Jul 06 20:24:00.931264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.931227 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb"] Jul 06 20:24:00.935286 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.935263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:00.938542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.938521 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 20:24:00.938647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.938544 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 20:24:00.938647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.938522 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sf5s6\"" Jul 06 20:24:00.938647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.938546 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 20:24:00.948653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.948630 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb"] Jul 06 20:24:00.961677 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:00.961652 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:01.011862 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.011836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.011985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.011869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.011985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.011884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.012069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.012028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.012069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.012054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.012133 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.012099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4598\" (UniqueName: \"kubernetes.io/projected/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-kube-api-access-z4598\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4598\" (UniqueName: \"kubernetes.io/projected/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-kube-api-access-z4598\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113537 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113772 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113746 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113832 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.113898 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.113883 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.115730 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.115705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.116045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.116026 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.120572 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.120552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4598\" (UniqueName: \"kubernetes.io/projected/63b1ec5a-b67b-4571-b65b-ce9e30d92e36-kube-api-access-z4598\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb\" (UID: \"63b1ec5a-b67b-4571-b65b-ce9e30d92e36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.248722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.248614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:01.411646 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.411613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb"] Jul 06 20:24:01.412578 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:24:01.412548 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b1ec5a_b67b_4571_b65b_ce9e30d92e36.slice/crio-73437de958a8fa83839c3bd7ef59d827692f34bef3fc00c589f0c332690d72c1 WatchSource:0}: Error finding container 73437de958a8fa83839c3bd7ef59d827692f34bef3fc00c589f0c332690d72c1: Status 404 returned error can't find the container with id 73437de958a8fa83839c3bd7ef59d827692f34bef3fc00c589f0c332690d72c1 Jul 06 20:24:01.781926 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:01.780753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" event={"ID":"63b1ec5a-b67b-4571-b65b-ce9e30d92e36","Type":"ContainerStarted","Data":"73437de958a8fa83839c3bd7ef59d827692f34bef3fc00c589f0c332690d72c1"} Jul 06 20:24:02.789548 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:02.789471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" event={"ID":"63b1ec5a-b67b-4571-b65b-ce9e30d92e36","Type":"ContainerStarted","Data":"41196323b96f54e6cbff4c9475ce66c6b608f663a80d3e1053a17e4e856535c6"} Jul 06 20:24:02.806748 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:02.806689 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" podStartSLOduration=1.693420448 podStartE2EDuration="2.806668517s" totalImagesPullingTime="1.113248069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:02.80642889 +0000 UTC m=+559.297052440" watchObservedRunningTime="2026-07-06 20:24:02.806668517 +0000 UTC m=+559.297292065" Jul 06 20:24:11.249640 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:11.249608 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:11.250033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:11.249662 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:11.262287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:11.262265 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:11.846841 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:11.846813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb" Jul 06 20:24:17.778521 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:17.778483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:31.382388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:31.382347 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:38.217524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:38.217486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:42.844002 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.843972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps"] Jul 06 20:24:42.846832 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.846811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.849146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.849127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 20:24:42.855963 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.855940 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps"] Jul 06 20:24:42.869036 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.869012 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:42.944439 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhxx\" (UniqueName: \"kubernetes.io/projected/f898e230-9f00-49e4-b750-a66ffb0297a1-kube-api-access-jzhxx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.944561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.944623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.944623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f898e230-9f00-49e4-b750-a66ffb0297a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.944623 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:42.944758 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:42.944683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.045954 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.045922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046107 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.045958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f898e230-9f00-49e4-b750-a66ffb0297a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046107 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.045978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046107 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046288 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhxx\" (UniqueName: \"kubernetes.io/projected/f898e230-9f00-49e4-b750-a66ffb0297a1-kube-api-access-jzhxx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046288 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046310 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046396 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.046590 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.046567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.048254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.048225 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f898e230-9f00-49e4-b750-a66ffb0297a1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.048630 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.048615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f898e230-9f00-49e4-b750-a66ffb0297a1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.055097 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.055075 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhxx\" (UniqueName: \"kubernetes.io/projected/f898e230-9f00-49e4-b750-a66ffb0297a1-kube-api-access-jzhxx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps\" (UID: \"f898e230-9f00-49e4-b750-a66ffb0297a1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.157169 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.157143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:43.293738 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.293710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps"] Jul 06 20:24:43.294699 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:24:43.294676 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf898e230_9f00_49e4_b750_a66ffb0297a1.slice/crio-e3a96920eb2663cc31983da71081a5c3d1db6601ad22f8681362496a555d75f4 WatchSource:0}: Error finding container e3a96920eb2663cc31983da71081a5c3d1db6601ad22f8681362496a555d75f4: Status 404 returned error can't find the container with id e3a96920eb2663cc31983da71081a5c3d1db6601ad22f8681362496a555d75f4 Jul 06 20:24:43.946787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.946762 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:24:43.947472 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:43.947455 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:24:44.006444 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:44.006413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" event={"ID":"f898e230-9f00-49e4-b750-a66ffb0297a1","Type":"ContainerStarted","Data":"b5e17db4bec11914280027b66bc2ccc26f0e9af3c0bcf36395f4dbdcf67cdd06"} Jul 06 20:24:44.006601 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:44.006491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" event={"ID":"f898e230-9f00-49e4-b750-a66ffb0297a1","Type":"ContainerStarted","Data":"e3a96920eb2663cc31983da71081a5c3d1db6601ad22f8681362496a555d75f4"} Jul 06 20:24:44.022698 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:44.022612 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" podStartSLOduration=1.842520163 podStartE2EDuration="2.022567788s" totalImagesPullingTime="180.047625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:24:44.022284817 +0000 UTC m=+600.512908367" watchObservedRunningTime="2026-07-06 20:24:44.022567788 +0000 UTC m=+600.513191348" Jul 06 20:24:47.325103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:47.325062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:53.158263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:53.158230 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:53.158763 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:53.158300 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:53.172145 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:53.172119 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:53.173024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:53.172981 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:24:54.077149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:54.077116 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps" Jul 06 20:24:55.039612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:24:55.039579 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:26:08.749340 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.749307 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cbbfdc888-gvdqk"] Jul 06 20:26:08.752304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.752289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.762062 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.762039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cbbfdc888-gvdqk"] Jul 06 20:26:08.799097 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.799069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f0fc03-9347-4fff-8753-6d65e0746a0c-tls-cert\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.799210 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.799117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4xr\" (UniqueName: \"kubernetes.io/projected/91f0fc03-9347-4fff-8753-6d65e0746a0c-kube-api-access-9g4xr\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.901775 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.901746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f0fc03-9347-4fff-8753-6d65e0746a0c-tls-cert\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.901897 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.901795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4xr\" (UniqueName: \"kubernetes.io/projected/91f0fc03-9347-4fff-8753-6d65e0746a0c-kube-api-access-9g4xr\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.904136 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.904111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f0fc03-9347-4fff-8753-6d65e0746a0c-tls-cert\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:08.910047 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:08.910027 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4xr\" (UniqueName: \"kubernetes.io/projected/91f0fc03-9347-4fff-8753-6d65e0746a0c-kube-api-access-9g4xr\") pod \"authorino-7cbbfdc888-gvdqk\" (UID: \"91f0fc03-9347-4fff-8753-6d65e0746a0c\") " pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:09.061859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.061806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" Jul 06 20:26:09.186394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.186359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:26:09.215228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.215207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cbbfdc888-gvdqk"] Jul 06 20:26:09.217427 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:26:09.217403 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f0fc03_9347_4fff_8753_6d65e0746a0c.slice/crio-9199ea67ba45bec4975febf36092657a9cafac44ac01cb920e0099210af32184 WatchSource:0}: Error finding container 9199ea67ba45bec4975febf36092657a9cafac44ac01cb920e0099210af32184: Status 404 returned error can't find the container with id 9199ea67ba45bec4975febf36092657a9cafac44ac01cb920e0099210af32184 Jul 06 20:26:09.218654 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.218637 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:26:09.314750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.314695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:26:09.461085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:09.461059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" event={"ID":"91f0fc03-9347-4fff-8753-6d65e0746a0c","Type":"ContainerStarted","Data":"9199ea67ba45bec4975febf36092657a9cafac44ac01cb920e0099210af32184"} Jul 06 20:26:10.468641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.468604 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" event={"ID":"91f0fc03-9347-4fff-8753-6d65e0746a0c","Type":"ContainerStarted","Data":"bd46c25bc1562f42259d2a2e615021ab590024d892f0dfa4a6288d057701c3f2"} Jul 06 20:26:10.528065 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.527986 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cbbfdc888-gvdqk" podStartSLOduration=1.978865741 podStartE2EDuration="2.527964282s" totalImagesPullingTime="549.098541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:10.519179312 +0000 UTC m=+687.009802862" watchObservedRunningTime="2026-07-06 20:26:10.527964282 +0000 UTC m=+687.018587833" Jul 06 20:26:10.535602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.535573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:26:10.610368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.610336 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:26:10.610535 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.610512 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-57884f4449-p568p" podUID="72ffe854-3810-488f-b756-7a2bd64fb4ad" containerName="authorino" containerID="cri-o://0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3" gracePeriod=30 Jul 06 20:26:10.867458 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.867432 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:26:10.898574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.898543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:26:10.919737 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.919709 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8dl6\" (UniqueName: \"kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6\") pod \"72ffe854-3810-488f-b756-7a2bd64fb4ad\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " Jul 06 20:26:10.919877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.919752 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert\") pod \"72ffe854-3810-488f-b756-7a2bd64fb4ad\" (UID: \"72ffe854-3810-488f-b756-7a2bd64fb4ad\") " Jul 06 20:26:10.921868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.921837 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6" (OuterVolumeSpecName: "kube-api-access-l8dl6") pod "72ffe854-3810-488f-b756-7a2bd64fb4ad" (UID: "72ffe854-3810-488f-b756-7a2bd64fb4ad"). InnerVolumeSpecName "kube-api-access-l8dl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:26:10.932696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:10.932661 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "72ffe854-3810-488f-b756-7a2bd64fb4ad" (UID: "72ffe854-3810-488f-b756-7a2bd64fb4ad"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:26:11.021160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.021082 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72ffe854-3810-488f-b756-7a2bd64fb4ad-tls-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:26:11.021160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.021119 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dl6\" (UniqueName: \"kubernetes.io/projected/72ffe854-3810-488f-b756-7a2bd64fb4ad-kube-api-access-l8dl6\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:26:11.475970 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.475929 2574 generic.go:356] "Generic (PLEG): container finished" podID="72ffe854-3810-488f-b756-7a2bd64fb4ad" containerID="0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3" exitCode=0 Jul 06 20:26:11.475970 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.475975 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57884f4449-p568p" Jul 06 20:26:11.476472 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.476017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57884f4449-p568p" event={"ID":"72ffe854-3810-488f-b756-7a2bd64fb4ad","Type":"ContainerDied","Data":"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3"} Jul 06 20:26:11.476472 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.476057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57884f4449-p568p" event={"ID":"72ffe854-3810-488f-b756-7a2bd64fb4ad","Type":"ContainerDied","Data":"2d41ac52632cb6da6ad0461813588541c81748e2502e611644219f997db34caa"} Jul 06 20:26:11.476472 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.476079 2574 scope.go:117] "RemoveContainer" containerID="0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3" Jul 06 20:26:11.485661 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.485643 2574 scope.go:117] "RemoveContainer" containerID="0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3" Jul 06 20:26:11.486014 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:26:11.485986 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3\": container with ID starting with 0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3 not found: ID does not exist" containerID="0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3" Jul 06 20:26:11.486133 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.486027 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3"} err="failed to get container status \"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3\": rpc error: code = NotFound desc = could not find container \"0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3\": container with ID starting with 0a3108d3f3a9f7018029d7d7e3d107826bbae8dcaabbd805a35c6236799146a3 not found: ID does not exist" Jul 06 20:26:11.529833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.529791 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:26:11.555785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:11.555756 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57884f4449-p568p"] Jul 06 20:26:12.071575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:26:12.071545 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ffe854-3810-488f-b756-7a2bd64fb4ad" path="/var/lib/kubelet/pods/72ffe854-3810-488f-b756-7a2bd64fb4ad/volumes" Jul 06 20:27:00.955382 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:00.955341 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:02.355969 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:02.355934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:03.907281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:03.907248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:03.975846 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:03.975804 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:20.960656 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:20.960621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:21.064748 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.064716 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:27:21.065385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.065338 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-d79b4b87b-59tql" podUID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" containerName="manager" containerID="cri-o://5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e" gracePeriod=30 Jul 06 20:27:21.314671 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.314647 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:27:21.425839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.425799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert\") pod \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " Jul 06 20:27:21.426001 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.425877 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdngl\" (UniqueName: \"kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl\") pod \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\" (UID: \"ac934f11-22e7-4f54-b7c3-a08e7a2e8231\") " Jul 06 20:27:21.427738 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.427711 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ac934f11-22e7-4f54-b7c3-a08e7a2e8231" (UID: "ac934f11-22e7-4f54-b7c3-a08e7a2e8231"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:27:21.427864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.427781 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl" (OuterVolumeSpecName: "kube-api-access-rdngl") pod "ac934f11-22e7-4f54-b7c3-a08e7a2e8231" (UID: "ac934f11-22e7-4f54-b7c3-a08e7a2e8231"). InnerVolumeSpecName "kube-api-access-rdngl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:27:21.527497 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.527432 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdngl\" (UniqueName: \"kubernetes.io/projected/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-kube-api-access-rdngl\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:27:21.527497 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.527464 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac934f11-22e7-4f54-b7c3-a08e7a2e8231-webhook-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:27:21.865716 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.865641 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" containerID="5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e" exitCode=0 Jul 06 20:27:21.865716 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.865683 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-59tql" Jul 06 20:27:21.865878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.865729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-59tql" event={"ID":"ac934f11-22e7-4f54-b7c3-a08e7a2e8231","Type":"ContainerDied","Data":"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e"} Jul 06 20:27:21.865878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.865781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-59tql" event={"ID":"ac934f11-22e7-4f54-b7c3-a08e7a2e8231","Type":"ContainerDied","Data":"573b37fd81ed240a98689de8ae72c044d7f44d6c3d9e3ce1d9a7ba00c42d1ee4"} Jul 06 20:27:21.865878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.865806 2574 scope.go:117] "RemoveContainer" containerID="5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e" Jul 06 20:27:21.875575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.875560 2574 scope.go:117] "RemoveContainer" containerID="5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e" Jul 06 20:27:21.875803 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:21.875782 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e\": container with ID starting with 5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e not found: ID does not exist" containerID="5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e" Jul 06 20:27:21.875862 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.875819 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e"} err="failed to get container status \"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e\": rpc error: code = NotFound desc = could not find container \"5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e\": container with ID starting with 5e905523e57fe8ab82598d12582e541dd52406d1d5a27851f9cd3e4941b2c11e not found: ID does not exist" Jul 06 20:27:21.896358 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.896338 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:27:21.902054 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:21.902033 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-59tql"] Jul 06 20:27:22.071667 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.071643 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" path="/var/lib/kubelet/pods/ac934f11-22e7-4f54-b7c3-a08e7a2e8231/volumes" Jul 06 20:27:22.617394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.617361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:22.754868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.754833 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d79b4b87b-vnphq"] Jul 06 20:27:22.755245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755232 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ffe854-3810-488f-b756-7a2bd64fb4ad" containerName="authorino" Jul 06 20:27:22.755289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755247 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ffe854-3810-488f-b756-7a2bd64fb4ad" containerName="authorino" Jul 06 20:27:22.755289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755257 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" containerName="manager" Jul 06 20:27:22.755289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755263 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" containerName="manager" Jul 06 20:27:22.755385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755325 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac934f11-22e7-4f54-b7c3-a08e7a2e8231" containerName="manager" Jul 06 20:27:22.755385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.755334 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ffe854-3810-488f-b756-7a2bd64fb4ad" containerName="authorino" Jul 06 20:27:22.759863 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.759845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.764153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.764135 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7vbpg\"" Jul 06 20:27:22.764267 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.764137 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:27:22.768035 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.768018 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:27:22.782734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.782710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-vnphq"] Jul 06 20:27:22.838321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.838301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gkc\" (UniqueName: \"kubernetes.io/projected/35db64d2-85c0-4c6f-99b7-69608b972e3b-kube-api-access-b8gkc\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.838427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.838334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35db64d2-85c0-4c6f-99b7-69608b972e3b-webhook-cert\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.939105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.939080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gkc\" (UniqueName: \"kubernetes.io/projected/35db64d2-85c0-4c6f-99b7-69608b972e3b-kube-api-access-b8gkc\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.939204 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.939115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35db64d2-85c0-4c6f-99b7-69608b972e3b-webhook-cert\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.941440 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.941416 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35db64d2-85c0-4c6f-99b7-69608b972e3b-webhook-cert\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:22.949228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:22.949208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gkc\" (UniqueName: \"kubernetes.io/projected/35db64d2-85c0-4c6f-99b7-69608b972e3b-kube-api-access-b8gkc\") pod \"maas-controller-d79b4b87b-vnphq\" (UID: \"35db64d2-85c0-4c6f-99b7-69608b972e3b\") " pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:23.069296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:23.069273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:23.221761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:23.221735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d79b4b87b-vnphq"] Jul 06 20:27:23.223763 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:27:23.223736 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35db64d2_85c0_4c6f_99b7_69608b972e3b.slice/crio-6398ebade5d9cf78399bd74431d2b396ce1afa72ced7ad82ac274d550090b1f6 WatchSource:0}: Error finding container 6398ebade5d9cf78399bd74431d2b396ce1afa72ced7ad82ac274d550090b1f6: Status 404 returned error can't find the container with id 6398ebade5d9cf78399bd74431d2b396ce1afa72ced7ad82ac274d550090b1f6 Jul 06 20:27:23.752313 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:23.752273 2574 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/opendatahub/maas-controller@sha256:b6c25a1c408168c9d3a0caa484106dd73d30e99d24dce380f360e9f6eb6a76a4" Jul 06 20:27:23.752456 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:23.752375 2574 kuberuntime_manager.go:1459] "Unhandled Error" err="container manager start failed in pod maas-controller-d79b4b87b-vnphq_opendatahub(35db64d2-85c0-4c6f-99b7-69608b972e3b): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 06 20:27:23.752456 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:23.752407 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/maas-controller-d79b4b87b-vnphq" podUID="35db64d2-85c0-4c6f-99b7-69608b972e3b" Jul 06 20:27:23.877769 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:23.877661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-vnphq" event={"ID":"35db64d2-85c0-4c6f-99b7-69608b972e3b","Type":"ContainerStarted","Data":"6398ebade5d9cf78399bd74431d2b396ce1afa72ced7ad82ac274d550090b1f6"} Jul 06 20:27:23.878146 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:23.878116 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/maas-controller@sha256:b6c25a1c408168c9d3a0caa484106dd73d30e99d24dce380f360e9f6eb6a76a4\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/maas-controller-d79b4b87b-vnphq" podUID="35db64d2-85c0-4c6f-99b7-69608b972e3b" Jul 06 20:27:24.883255 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:27:24.883213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/maas-controller@sha256:b6c25a1c408168c9d3a0caa484106dd73d30e99d24dce380f360e9f6eb6a76a4\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/maas-controller-d79b4b87b-vnphq" podUID="35db64d2-85c0-4c6f-99b7-69608b972e3b" Jul 06 20:27:37.957003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:37.956969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d79b4b87b-vnphq" event={"ID":"35db64d2-85c0-4c6f-99b7-69608b972e3b","Type":"ContainerStarted","Data":"449d8a7840241b1002544a389a2cd261b2aae44212e234a49b25abc4041de545"} Jul 06 20:27:37.957380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:37.957166 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:27:37.992371 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:37.992329 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d79b4b87b-vnphq" podStartSLOduration=1.815100012 podStartE2EDuration="15.992315459s" totalImagesPullingTime="14.177215447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-06 20:27:37.991874751 +0000 UTC m=+774.482498302" watchObservedRunningTime="2026-07-06 20:27:37.992315459 +0000 UTC m=+774.482939015" Jul 06 20:27:48.153987 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:48.153934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:27:48.970377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:27:48.970344 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d79b4b87b-vnphq" Jul 06 20:28:15.507493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:15.507404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:15.719620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:15.719577 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:15.926551 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:15.926513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:23.218177 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:23.218146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:23.352372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:23.352345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:23.457307 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:23.457281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:23.562862 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:23.562786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:29.527103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:29.527072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:28:29.627930 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:28:29.627887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:29:43.980160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:29:43.980079 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:29:43.984024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:29:43.984000 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:30:00.148228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.148188 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-2tjsd"] Jul 06 20:30:00.151957 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.151936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:00.167624 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.167597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-2tjsd"] Jul 06 20:30:00.218094 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.218053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rd99\" (UniqueName: \"kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99\") pod \"maas-api-key-cleanup-29722830-2tjsd\" (UID: \"6314cea7-9e22-41e0-bcc9-640df1c4cc5f\") " pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:00.319734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.319689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rd99\" (UniqueName: \"kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99\") pod \"maas-api-key-cleanup-29722830-2tjsd\" (UID: \"6314cea7-9e22-41e0-bcc9-640df1c4cc5f\") " pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:00.328158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.328131 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rd99\" (UniqueName: \"kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99\") pod \"maas-api-key-cleanup-29722830-2tjsd\" (UID: \"6314cea7-9e22-41e0-bcc9-640df1c4cc5f\") " pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:00.462040 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.462008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:00.802678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:00.802652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-2tjsd"] Jul 06 20:30:00.804864 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:30:00.804833 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6314cea7_9e22_41e0_bcc9_640df1c4cc5f.slice/crio-a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870 WatchSource:0}: Error finding container a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870: Status 404 returned error can't find the container with id a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870 Jul 06 20:30:01.731884 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:01.731847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" event={"ID":"6314cea7-9e22-41e0-bcc9-640df1c4cc5f","Type":"ContainerStarted","Data":"a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870"} Jul 06 20:30:04.473232 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:04.473199 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:04.749359 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:04.749271 2574 generic.go:356] "Generic (PLEG): container finished" podID="6314cea7-9e22-41e0-bcc9-640df1c4cc5f" containerID="c4f2a1de16b417022866efde82fd607c7fba6c68666d3910f102099dd32af26b" exitCode=0 Jul 06 20:30:04.749510 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:04.749360 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" event={"ID":"6314cea7-9e22-41e0-bcc9-640df1c4cc5f","Type":"ContainerDied","Data":"c4f2a1de16b417022866efde82fd607c7fba6c68666d3910f102099dd32af26b"} Jul 06 20:30:05.891497 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:05.891466 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:05.972068 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:05.972045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rd99\" (UniqueName: \"kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99\") pod \"6314cea7-9e22-41e0-bcc9-640df1c4cc5f\" (UID: \"6314cea7-9e22-41e0-bcc9-640df1c4cc5f\") " Jul 06 20:30:05.974013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:05.973979 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99" (OuterVolumeSpecName: "kube-api-access-9rd99") pod "6314cea7-9e22-41e0-bcc9-640df1c4cc5f" (UID: "6314cea7-9e22-41e0-bcc9-640df1c4cc5f"). InnerVolumeSpecName "kube-api-access-9rd99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:06.072733 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:06.072681 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rd99\" (UniqueName: \"kubernetes.io/projected/6314cea7-9e22-41e0-bcc9-640df1c4cc5f-kube-api-access-9rd99\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:30:06.760774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:06.760748 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" Jul 06 20:30:06.760955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:06.760781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-2tjsd" event={"ID":"6314cea7-9e22-41e0-bcc9-640df1c4cc5f","Type":"ContainerDied","Data":"a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870"} Jul 06 20:30:06.760955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:06.760821 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86f17aa6473024f9312ece6462bfb1fa9170d2ec2b601c9782659c7024c7870" Jul 06 20:30:11.281380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:11.281340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:30:11.374951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:30:11.374896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:31:59.702438 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:31:59.702400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:32:28.800650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:28.800615 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:32:28.801046 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:28.800830 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" podUID="99c1d814-92d6-424c-8471-c7942a793415" containerName="manager" containerID="cri-o://d04914cf9f47d196d9317a1fff113de27e519e8614f8c1e97e5bb2ea0f482db9" gracePeriod=10 Jul 06 20:32:31.584579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.584550 2574 generic.go:356] "Generic (PLEG): container finished" podID="99c1d814-92d6-424c-8471-c7942a793415" containerID="d04914cf9f47d196d9317a1fff113de27e519e8614f8c1e97e5bb2ea0f482db9" exitCode=0 Jul 06 20:32:31.584952 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.584620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" event={"ID":"99c1d814-92d6-424c-8471-c7942a793415","Type":"ContainerDied","Data":"d04914cf9f47d196d9317a1fff113de27e519e8614f8c1e97e5bb2ea0f482db9"} Jul 06 20:32:31.661348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.661326 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:32:31.772239 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.772164 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume\") pod \"99c1d814-92d6-424c-8471-c7942a793415\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " Jul 06 20:32:31.772374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.772266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmx2s\" (UniqueName: \"kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s\") pod \"99c1d814-92d6-424c-8471-c7942a793415\" (UID: \"99c1d814-92d6-424c-8471-c7942a793415\") " Jul 06 20:32:31.772525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.772502 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "99c1d814-92d6-424c-8471-c7942a793415" (UID: "99c1d814-92d6-424c-8471-c7942a793415"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:32:31.774196 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.774177 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s" (OuterVolumeSpecName: "kube-api-access-nmx2s") pod "99c1d814-92d6-424c-8471-c7942a793415" (UID: "99c1d814-92d6-424c-8471-c7942a793415"). InnerVolumeSpecName "kube-api-access-nmx2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:31.873262 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.873239 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmx2s\" (UniqueName: \"kubernetes.io/projected/99c1d814-92d6-424c-8471-c7942a793415-kube-api-access-nmx2s\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:32:31.873262 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:31.873258 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/99c1d814-92d6-424c-8471-c7942a793415-extensions-socket-volume\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:32:32.591558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:32.591522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" event={"ID":"99c1d814-92d6-424c-8471-c7942a793415","Type":"ContainerDied","Data":"f15d902c76baebcb02c77b88121ade99c2acbae8907fde40a9ac6dd1eb6afc3f"} Jul 06 20:32:32.591558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:32.591541 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb" Jul 06 20:32:32.592101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:32.591604 2574 scope.go:117] "RemoveContainer" containerID="d04914cf9f47d196d9317a1fff113de27e519e8614f8c1e97e5bb2ea0f482db9" Jul 06 20:32:32.610542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:32.610521 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:32:32.617208 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:32.617185 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4bnjb"] Jul 06 20:32:34.072249 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:32:34.072216 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c1d814-92d6-424c-8471-c7942a793415" path="/var/lib/kubelet/pods/99c1d814-92d6-424c-8471-c7942a793415/volumes" Jul 06 20:33:29.920989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.920956 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz"] Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921315 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c1d814-92d6-424c-8471-c7942a793415" containerName="manager" Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921326 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c1d814-92d6-424c-8471-c7942a793415" containerName="manager" Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921353 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6314cea7-9e22-41e0-bcc9-640df1c4cc5f" containerName="cleanup" Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921359 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6314cea7-9e22-41e0-bcc9-640df1c4cc5f" containerName="cleanup" Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921428 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c1d814-92d6-424c-8471-c7942a793415" containerName="manager" Jul 06 20:33:29.921443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.921438 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6314cea7-9e22-41e0-bcc9-640df1c4cc5f" containerName="cleanup" Jul 06 20:33:29.923505 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.923488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:29.925761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.925739 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kcwqt\"" Jul 06 20:33:29.925979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.925898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4469b32d-138a-43fb-abc2-4b60b8e75adc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:29.926034 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.926020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcgx\" (UniqueName: \"kubernetes.io/projected/4469b32d-138a-43fb-abc2-4b60b8e75adc-kube-api-access-4wcgx\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:29.933970 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:29.933948 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz"] Jul 06 20:33:30.026797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.026764 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4469b32d-138a-43fb-abc2-4b60b8e75adc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.026974 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.026812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcgx\" (UniqueName: \"kubernetes.io/projected/4469b32d-138a-43fb-abc2-4b60b8e75adc-kube-api-access-4wcgx\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.027144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.027124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4469b32d-138a-43fb-abc2-4b60b8e75adc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.035038 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.035019 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcgx\" (UniqueName: \"kubernetes.io/projected/4469b32d-138a-43fb-abc2-4b60b8e75adc-kube-api-access-4wcgx\") pod \"kuadrant-operator-controller-manager-86878c769f-2hzfz\" (UID: \"4469b32d-138a-43fb-abc2-4b60b8e75adc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.234196 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.234122 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.579075 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.579048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz"] Jul 06 20:33:30.581310 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:33:30.581286 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4469b32d_138a_43fb_abc2_4b60b8e75adc.slice/crio-f15d96a0e22b02be5079bfbfefd5a5088aefda33ba7c3f27b441203d6f562c25 WatchSource:0}: Error finding container f15d96a0e22b02be5079bfbfefd5a5088aefda33ba7c3f27b441203d6f562c25: Status 404 returned error can't find the container with id f15d96a0e22b02be5079bfbfefd5a5088aefda33ba7c3f27b441203d6f562c25 Jul 06 20:33:30.901309 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.901235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" event={"ID":"4469b32d-138a-43fb-abc2-4b60b8e75adc","Type":"ContainerStarted","Data":"d7a22b72727c62042876e7bbe49dcef7d801c2621418b7b647de1655538a0d0e"} Jul 06 20:33:30.901309 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.901288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" event={"ID":"4469b32d-138a-43fb-abc2-4b60b8e75adc","Type":"ContainerStarted","Data":"f15d96a0e22b02be5079bfbfefd5a5088aefda33ba7c3f27b441203d6f562c25"} Jul 06 20:33:30.901488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.901327 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:33:30.921770 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:30.921726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" podStartSLOduration=1.921713413 podStartE2EDuration="1.921713413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:30.919331824 +0000 UTC m=+1127.409955374" watchObservedRunningTime="2026-07-06 20:33:30.921713413 +0000 UTC m=+1127.412336962" Jul 06 20:33:41.909045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:33:41.909011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2hzfz" Jul 06 20:34:35.367126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:34:35.367090 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:36.826064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:34:36.826026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:34:44.014336 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:34:44.014305 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:34:44.019090 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:34:44.019065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:37:12.339086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:37:12.339006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:05.256489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:05.256411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:08.225923 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:08.225870 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:13.907036 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:13.906984 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:21.373086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:21.373048 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:29.650316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:29.650278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:34.784388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:34.784352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:37.668499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:37.668461 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:39.024292 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:39.024252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:43.932224 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:43.932179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.002556 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.002530 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.051081 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.051057 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:39:44.057922 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.057880 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:39:44.124414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.124381 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:44.127579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.127561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.130149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.130126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3f62c2e8-openshift-default-dockercfg-xpmr6\"" Jul 06 20:39:44.150352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.150309 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:44.168204 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.168180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.201515 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201515 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201668 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201668 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88w5z\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201668 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.201831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.201815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.264877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.264857 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:44.302384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w5z\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.302707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.303002 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302899 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.303002 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.302949 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.303141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.303119 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.303193 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.303153 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.303440 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.303416 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.304883 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.304864 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.305179 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.305160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.310892 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.310867 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w5z\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.311149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.311129 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token\") pod \"e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.437791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.437755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:44.579620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.579595 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:44.581661 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:39:44.581636 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c30908e_6660_4b87_8c63_50c0cd86b537.slice/crio-d5f0216b030ef9ab991a27a19c2459b4096ad87fdac1604633afe3a3c94a4259 WatchSource:0}: Error finding container d5f0216b030ef9ab991a27a19c2459b4096ad87fdac1604633afe3a3c94a4259: Status 404 returned error can't find the container with id d5f0216b030ef9ab991a27a19c2459b4096ad87fdac1604633afe3a3c94a4259 Jul 06 20:39:44.584314 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.584262 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:44.584423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.584363 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:44.584423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.584409 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:44.988754 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.988718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" event={"ID":"5c30908e-6660-4b87-8c63-50c0cd86b537","Type":"ContainerStarted","Data":"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4"} Jul 06 20:39:44.989249 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:44.988778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" event={"ID":"5c30908e-6660-4b87-8c63-50c0cd86b537","Type":"ContainerStarted","Data":"d5f0216b030ef9ab991a27a19c2459b4096ad87fdac1604633afe3a3c94a4259"} Jul 06 20:39:45.010385 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.010320 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" podStartSLOduration=1.010271324 podStartE2EDuration="1.010271324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:45.009973312 +0000 UTC m=+1501.500596857" watchObservedRunningTime="2026-07-06 20:39:45.010271324 +0000 UTC m=+1501.500894871" Jul 06 20:39:45.361081 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.360992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:45.421078 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.421048 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:45.423531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.423512 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.425800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.425776 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-3f62c2e8-serving-cert\"" Jul 06 20:39:45.437459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.437438 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:45.438277 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.438251 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:45.443831 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.443810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:45.468226 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.468206 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:45.513337 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.513316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.513439 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.513423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lqfv\" (UniqueName: \"kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.513483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.513450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.573137 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.573109 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:45.614451 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.614382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lqfv\" (UniqueName: \"kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.614451 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.614411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.614625 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.614452 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.615058 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.615042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.616960 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.616938 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.627218 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.627199 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lqfv\" (UniqueName: \"kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv\") pod \"maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.669281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.669257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:45.734655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.734632 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:45.994425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.994396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:45.995412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:45.995389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:46.089505 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:46.089477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:46.091820 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:39:46.091795 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141ff636_5d23_47ed_8be4_cdb796965626.slice/crio-25a6d11cef5fac549725ad17ecef6ab3bebbe9037e8bb8c0815a9f8bfbf5319f WatchSource:0}: Error finding container 25a6d11cef5fac549725ad17ecef6ab3bebbe9037e8bb8c0815a9f8bfbf5319f: Status 404 returned error can't find the container with id 25a6d11cef5fac549725ad17ecef6ab3bebbe9037e8bb8c0815a9f8bfbf5319f Jul 06 20:39:46.093511 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:46.093488 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:39:46.170712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:46.170684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:46.279091 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:46.279006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:47.002067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:47.002032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" event={"ID":"141ff636-5d23-47ed-8be4-cdb796965626","Type":"ContainerStarted","Data":"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc"} Jul 06 20:39:47.002561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:47.002090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" event={"ID":"141ff636-5d23-47ed-8be4-cdb796965626","Type":"ContainerStarted","Data":"25a6d11cef5fac549725ad17ecef6ab3bebbe9037e8bb8c0815a9f8bfbf5319f"} Jul 06 20:39:47.002561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:47.002266 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:47.033591 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:47.033550 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" podStartSLOduration=1.550378769 podStartE2EDuration="2.033535367s" totalImagesPullingTime="483.156598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:47.030025182 +0000 UTC m=+1503.520648732" watchObservedRunningTime="2026-07-06 20:39:47.033535367 +0000 UTC m=+1503.524158948" Jul 06 20:39:47.846329 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:47.846294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.391644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.391603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.551696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.551659 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:50.551961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.551892 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" podUID="5c30908e-6660-4b87-8c63-50c0cd86b537" containerName="istio-proxy" containerID="cri-o://1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4" gracePeriod=30 Jul 06 20:39:50.590754 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.590727 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.676024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.675994 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:50.754355 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.754323 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:50.754565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.754540 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" podUID="141ff636-5d23-47ed-8be4-cdb796965626" containerName="maas-api" containerID="cri-o://f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc" gracePeriod=30 Jul 06 20:39:50.784211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:50.784043 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:51.003961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.003940 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:51.026583 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.026555 2574 generic.go:356] "Generic (PLEG): container finished" podID="141ff636-5d23-47ed-8be4-cdb796965626" containerID="f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc" exitCode=0 Jul 06 20:39:51.026693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.026601 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" Jul 06 20:39:51.026693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.026638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" event={"ID":"141ff636-5d23-47ed-8be4-cdb796965626","Type":"ContainerDied","Data":"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc"} Jul 06 20:39:51.026776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.026701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc" event={"ID":"141ff636-5d23-47ed-8be4-cdb796965626","Type":"ContainerDied","Data":"25a6d11cef5fac549725ad17ecef6ab3bebbe9037e8bb8c0815a9f8bfbf5319f"} Jul 06 20:39:51.026776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.026737 2574 scope.go:117] "RemoveContainer" containerID="f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc" Jul 06 20:39:51.037127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.037111 2574 scope.go:117] "RemoveContainer" containerID="f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc" Jul 06 20:39:51.037394 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:39:51.037377 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc\": container with ID starting with f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc not found: ID does not exist" containerID="f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc" Jul 06 20:39:51.037447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.037402 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc"} err="failed to get container status \"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc\": rpc error: code = NotFound desc = could not find container \"f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc\": container with ID starting with f6436c85b3194709da8264ac9ad56a5dc139d4a78ad45d4c62e716d2360a9fcc not found: ID does not exist" Jul 06 20:39:51.060776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.060754 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca\") pod \"141ff636-5d23-47ed-8be4-cdb796965626\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " Jul 06 20:39:51.060880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.060783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls\") pod \"141ff636-5d23-47ed-8be4-cdb796965626\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " Jul 06 20:39:51.060880 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.060805 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lqfv\" (UniqueName: \"kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv\") pod \"141ff636-5d23-47ed-8be4-cdb796965626\" (UID: \"141ff636-5d23-47ed-8be4-cdb796965626\") " Jul 06 20:39:51.061149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.061128 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "141ff636-5d23-47ed-8be4-cdb796965626" (UID: "141ff636-5d23-47ed-8be4-cdb796965626"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:51.062763 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.062734 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv" (OuterVolumeSpecName: "kube-api-access-6lqfv") pod "141ff636-5d23-47ed-8be4-cdb796965626" (UID: "141ff636-5d23-47ed-8be4-cdb796965626"). InnerVolumeSpecName "kube-api-access-6lqfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:51.062829 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.062760 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "141ff636-5d23-47ed-8be4-cdb796965626" (UID: "141ff636-5d23-47ed-8be4-cdb796965626"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:39:51.162118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.162092 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141ff636-5d23-47ed-8be4-cdb796965626-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:51.162118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.162115 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141ff636-5d23-47ed-8be4-cdb796965626-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:51.162264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.162125 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lqfv\" (UniqueName: \"kubernetes.io/projected/141ff636-5d23-47ed-8be4-cdb796965626-kube-api-access-6lqfv\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:51.348887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.348862 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:51.353406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:51.353381 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-3f62c2e8-c8f778c5f-w68jc"] Jul 06 20:39:52.072333 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:52.072300 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141ff636-5d23-47ed-8be4-cdb796965626" path="/var/lib/kubelet/pods/141ff636-5d23-47ed-8be4-cdb796965626/volumes" Jul 06 20:39:52.323119 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:52.323042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:55.810348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.810326 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:55.900493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900436 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900518 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88w5z\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900536 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900575 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900590 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900613 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.900688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900649 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket\") pod \"5c30908e-6660-4b87-8c63-50c0cd86b537\" (UID: \"5c30908e-6660-4b87-8c63-50c0cd86b537\") " Jul 06 20:39:55.901071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900730 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:39:55.901071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.900898 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:39:55.901071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.901021 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:39:55.901217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.901074 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:55.901217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.901094 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:55.901217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.901157 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:39:55.901217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.901195 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:39:55.902720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.902687 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:55.902879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.902859 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:39:55.903068 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.903053 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z" (OuterVolumeSpecName: "kube-api-access-88w5z") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "kube-api-access-88w5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:39:55.903136 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:55.903114 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c30908e-6660-4b87-8c63-50c0cd86b537" (UID: "5c30908e-6660-4b87-8c63-50c0cd86b537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:39:56.001760 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001735 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30908e-6660-4b87-8c63-50c0cd86b537-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001760 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001756 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001760 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001764 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001773 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001782 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88w5z\" (UniqueName: \"kubernetes.io/projected/5c30908e-6660-4b87-8c63-50c0cd86b537-kube-api-access-88w5z\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001791 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30908e-6660-4b87-8c63-50c0cd86b537-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.001946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.001799 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30908e-6660-4b87-8c63-50c0cd86b537-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:39:56.057678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.057648 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c30908e-6660-4b87-8c63-50c0cd86b537" containerID="1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4" exitCode=0 Jul 06 20:39:56.057761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.057692 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" Jul 06 20:39:56.057761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.057700 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" event={"ID":"5c30908e-6660-4b87-8c63-50c0cd86b537","Type":"ContainerDied","Data":"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4"} Jul 06 20:39:56.057761 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.057744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k" event={"ID":"5c30908e-6660-4b87-8c63-50c0cd86b537","Type":"ContainerDied","Data":"d5f0216b030ef9ab991a27a19c2459b4096ad87fdac1604633afe3a3c94a4259"} Jul 06 20:39:56.057868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.057769 2574 scope.go:117] "RemoveContainer" containerID="1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4" Jul 06 20:39:56.066718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.066701 2574 scope.go:117] "RemoveContainer" containerID="1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4" Jul 06 20:39:56.066996 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:39:56.066975 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4\": container with ID starting with 1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4 not found: ID does not exist" containerID="1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4" Jul 06 20:39:56.067059 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.067005 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4"} err="failed to get container status \"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4\": rpc error: code = NotFound desc = could not find container \"1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4\": container with ID starting with 1ebfb3ccaa5e691f9b9aa09fcc4dd1ebff01a9959ceb8e8ea5f1781f5f967ae4 not found: ID does not exist" Jul 06 20:39:56.081311 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.081286 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:56.087534 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.087514 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3f62c2e8-openshift-default-8496f96fd9-jzx5k"] Jul 06 20:39:56.857499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.857463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:56.962951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:56.962866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:57.035495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035466 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:39:57.035875 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035859 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141ff636-5d23-47ed-8be4-cdb796965626" containerName="maas-api" Jul 06 20:39:57.035875 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035875 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="141ff636-5d23-47ed-8be4-cdb796965626" containerName="maas-api" Jul 06 20:39:57.036066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035897 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c30908e-6660-4b87-8c63-50c0cd86b537" containerName="istio-proxy" Jul 06 20:39:57.036066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035930 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c30908e-6660-4b87-8c63-50c0cd86b537" containerName="istio-proxy" Jul 06 20:39:57.036066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.035991 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="141ff636-5d23-47ed-8be4-cdb796965626" containerName="maas-api" Jul 06 20:39:57.036066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.036003 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c30908e-6660-4b87-8c63-50c0cd86b537" containerName="istio-proxy" Jul 06 20:39:57.044685 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.044664 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.048066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.048048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a7843468-openshift-default-dockercfg-gf67s\"" Jul 06 20:39:57.076192 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.075834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:39:57.111818 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.111818 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.111818 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vwx\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.111975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.112221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.112080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.156069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.156038 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:57.213162 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213382 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213382 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213498 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213498 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vwx\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213598 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213598 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213598 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213554 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213756 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213756 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213731 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.213955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.213935 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.215506 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.215487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.215856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.215838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.225920 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.225889 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vwx\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.232145 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.232123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token\") pod \"e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.298336 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.298312 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:57.356070 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.356049 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:57.500245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.500214 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:39:57.501069 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:39:57.501037 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0d84cc_e75c_4b61_b2ae_2c629564cda1.slice/crio-cf39c350f9a7f9ec00694a500ad6a793a2f15c5556e8ddf2485ef45765ba8c2e WatchSource:0}: Error finding container cf39c350f9a7f9ec00694a500ad6a793a2f15c5556e8ddf2485ef45765ba8c2e: Status 404 returned error can't find the container with id cf39c350f9a7f9ec00694a500ad6a793a2f15c5556e8ddf2485ef45765ba8c2e Jul 06 20:39:57.503159 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.503127 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:57.503243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.503188 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:57.503243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:57.503232 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:39:58.073436 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.073403 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c30908e-6660-4b87-8c63-50c0cd86b537" path="/var/lib/kubelet/pods/5c30908e-6660-4b87-8c63-50c0cd86b537/volumes" Jul 06 20:39:58.073788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.073706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" event={"ID":"ee0d84cc-e75c-4b61-b2ae-2c629564cda1","Type":"ContainerStarted","Data":"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0"} Jul 06 20:39:58.073788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.073743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" event={"ID":"ee0d84cc-e75c-4b61-b2ae-2c629564cda1","Type":"ContainerStarted","Data":"cf39c350f9a7f9ec00694a500ad6a793a2f15c5556e8ddf2485ef45765ba8c2e"} Jul 06 20:39:58.101491 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.101444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" podStartSLOduration=1.101428178 podStartE2EDuration="1.101428178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:39:58.100487553 +0000 UTC m=+1514.591111096" watchObservedRunningTime="2026-07-06 20:39:58.101428178 +0000 UTC m=+1514.592051727" Jul 06 20:39:58.289297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.289264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:58.357025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.356949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:58.362888 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.362860 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:58.370772 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.370742 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:39:58.374275 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.374252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.376424 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.376402 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a7843468-serving-cert\"" Jul 06 20:39:58.389149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.389126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:39:58.402864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.402839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:58.425788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.425769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.425894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.425811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.425981 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.425950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4dt\" (UniqueName: \"kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.502392 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.502361 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:58.527484 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.527441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.527658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.527527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.527658 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:39:58.527561 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-a7843468-serving-cert: secret "maas-api-e2e-ait-a7843468-serving-cert" not found Jul 06 20:39:58.527658 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:39:58.527635 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls podName:360e8302-f87f-456f-b8fd-6e02615c8b79 nodeName:}" failed. No retries permitted until 2026-07-06 20:39:59.027611528 +0000 UTC m=+1515.518235068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls") pod "maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" (UID: "360e8302-f87f-456f-b8fd-6e02615c8b79") : secret "maas-api-e2e-ait-a7843468-serving-cert" not found Jul 06 20:39:58.527833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.527680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4dt\" (UniqueName: \"kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.528344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.528324 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.537050 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.537029 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4dt\" (UniqueName: \"kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:58.707435 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:58.707408 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:59.032362 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.032285 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:59.038462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.038436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:59.040887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.040857 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") pod \"maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:59.079503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.079479 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:59.080441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.080424 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:39:59.115016 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.114992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:39:59.288022 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.287896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:39:59.426788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:39:59.426761 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:39:59.427848 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:39:59.427822 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360e8302_f87f_456f_b8fd_6e02615c8b79.slice/crio-39f6be02ea87079374b9bf3e233da48369dd3716824c265539775fea9bdf0922 WatchSource:0}: Error finding container 39f6be02ea87079374b9bf3e233da48369dd3716824c265539775fea9bdf0922: Status 404 returned error can't find the container with id 39f6be02ea87079374b9bf3e233da48369dd3716824c265539775fea9bdf0922 Jul 06 20:40:00.087606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:00.087501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" event={"ID":"360e8302-f87f-456f-b8fd-6e02615c8b79","Type":"ContainerStarted","Data":"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436"} Jul 06 20:40:00.087606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:00.087564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" event={"ID":"360e8302-f87f-456f-b8fd-6e02615c8b79","Type":"ContainerStarted","Data":"39f6be02ea87079374b9bf3e233da48369dd3716824c265539775fea9bdf0922"} Jul 06 20:40:00.087606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:00.087600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:40:00.111318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:00.111264 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" podStartSLOduration=1.763685058 podStartE2EDuration="2.111247419s" totalImagesPullingTime="347.562361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:00.109296206 +0000 UTC m=+1516.599919752" watchObservedRunningTime="2026-07-06 20:40:00.111247419 +0000 UTC m=+1516.601870973" Jul 06 20:40:00.385564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:00.385532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:03.298028 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:03.297997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:03.683927 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:03.681491 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:40:03.683927 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:03.681727 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" podUID="360e8302-f87f-456f-b8fd-6e02615c8b79" containerName="maas-api" containerID="cri-o://e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436" gracePeriod=30 Jul 06 20:40:03.703928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:03.702888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:03.964940 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:03.964917 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:40:04.092955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.091247 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n4dt\" (UniqueName: \"kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt\") pod \"360e8302-f87f-456f-b8fd-6e02615c8b79\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " Jul 06 20:40:04.092955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.091308 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca\") pod \"360e8302-f87f-456f-b8fd-6e02615c8b79\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " Jul 06 20:40:04.092955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.091394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") pod \"360e8302-f87f-456f-b8fd-6e02615c8b79\" (UID: \"360e8302-f87f-456f-b8fd-6e02615c8b79\") " Jul 06 20:40:04.092955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.092472 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "360e8302-f87f-456f-b8fd-6e02615c8b79" (UID: "360e8302-f87f-456f-b8fd-6e02615c8b79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:40:04.097927 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.095373 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt" (OuterVolumeSpecName: "kube-api-access-6n4dt") pod "360e8302-f87f-456f-b8fd-6e02615c8b79" (UID: "360e8302-f87f-456f-b8fd-6e02615c8b79"). InnerVolumeSpecName "kube-api-access-6n4dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:04.107518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.107486 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "360e8302-f87f-456f-b8fd-6e02615c8b79" (UID: "360e8302-f87f-456f-b8fd-6e02615c8b79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:40:04.112412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.112385 2574 generic.go:356] "Generic (PLEG): container finished" podID="360e8302-f87f-456f-b8fd-6e02615c8b79" containerID="e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436" exitCode=0 Jul 06 20:40:04.112521 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.112434 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" Jul 06 20:40:04.112521 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.112483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" event={"ID":"360e8302-f87f-456f-b8fd-6e02615c8b79","Type":"ContainerDied","Data":"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436"} Jul 06 20:40:04.112626 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.112548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7" event={"ID":"360e8302-f87f-456f-b8fd-6e02615c8b79","Type":"ContainerDied","Data":"39f6be02ea87079374b9bf3e233da48369dd3716824c265539775fea9bdf0922"} Jul 06 20:40:04.112626 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.112581 2574 scope.go:117] "RemoveContainer" containerID="e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436" Jul 06 20:40:04.124378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.124356 2574 scope.go:117] "RemoveContainer" containerID="e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436" Jul 06 20:40:04.124632 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:40:04.124611 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436\": container with ID starting with e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436 not found: ID does not exist" containerID="e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436" Jul 06 20:40:04.124694 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.124642 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436"} err="failed to get container status \"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436\": rpc error: code = NotFound desc = could not find container \"e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436\": container with ID starting with e41819b08768af85c0112a6f6746e3970a3372d911cebcabe43911e52c53a436 not found: ID does not exist" Jul 06 20:40:04.145093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.145068 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:40:04.150533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.150513 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a7843468-66d8fc6d8c-zskv7"] Jul 06 20:40:04.204011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.192549 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4dt\" (UniqueName: \"kubernetes.io/projected/360e8302-f87f-456f-b8fd-6e02615c8b79-kube-api-access-6n4dt\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:04.204011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.192586 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/360e8302-f87f-456f-b8fd-6e02615c8b79-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:04.204011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:04.192602 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/360e8302-f87f-456f-b8fd-6e02615c8b79-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:06.071877 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:06.071803 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="360e8302-f87f-456f-b8fd-6e02615c8b79" path="/var/lib/kubelet/pods/360e8302-f87f-456f-b8fd-6e02615c8b79/volumes" Jul 06 20:40:09.374475 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:09.374443 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:40:09.375101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:09.374632 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" podUID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" containerName="istio-proxy" containerID="cri-o://e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0" gracePeriod=30 Jul 06 20:40:09.399435 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:09.399405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:09.504157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:09.504120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:10.623942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.623869 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:40:10.753843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753755 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.753843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.753843 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753870 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753932 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.753978 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754032 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68vwx\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754067 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo\") pod \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\" (UID: \"ee0d84cc-e75c-4b61-b2ae-2c629564cda1\") " Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754089 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:10.754121 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754107 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:10.754452 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754193 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:10.754452 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754418 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:40:10.754696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754660 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.754696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754686 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.754825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754702 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.754825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754717 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.754825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.754738 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:10.756273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.756240 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx" (OuterVolumeSpecName: "kube-api-access-68vwx") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "kube-api-access-68vwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:10.756273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.756261 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:10.756410 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.756254 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:40:10.756410 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.756298 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee0d84cc-e75c-4b61-b2ae-2c629564cda1" (UID: "ee0d84cc-e75c-4b61-b2ae-2c629564cda1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:10.856118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.856094 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.856118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.856114 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.856244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.856123 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vwx\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-kube-api-access-68vwx\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.856244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.856132 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:10.856244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:10.856141 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee0d84cc-e75c-4b61-b2ae-2c629564cda1-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:11.156174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.156141 2574 generic.go:356] "Generic (PLEG): container finished" podID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" containerID="e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0" exitCode=0 Jul 06 20:40:11.156318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.156192 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" Jul 06 20:40:11.156318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.156230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" event={"ID":"ee0d84cc-e75c-4b61-b2ae-2c629564cda1","Type":"ContainerDied","Data":"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0"} Jul 06 20:40:11.156318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.156286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv" event={"ID":"ee0d84cc-e75c-4b61-b2ae-2c629564cda1","Type":"ContainerDied","Data":"cf39c350f9a7f9ec00694a500ad6a793a2f15c5556e8ddf2485ef45765ba8c2e"} Jul 06 20:40:11.156318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.156308 2574 scope.go:117] "RemoveContainer" containerID="e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0" Jul 06 20:40:11.166116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.166100 2574 scope.go:117] "RemoveContainer" containerID="e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0" Jul 06 20:40:11.166343 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:40:11.166326 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0\": container with ID starting with e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0 not found: ID does not exist" containerID="e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0" Jul 06 20:40:11.166389 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.166352 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0"} err="failed to get container status \"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0\": rpc error: code = NotFound desc = could not find container \"e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0\": container with ID starting with e8716688aa26e26ee0fed6017654c2ff87fb7dfda56096834787e8389b8c48e0 not found: ID does not exist" Jul 06 20:40:11.178646 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.178611 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:40:11.182503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:11.182480 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a7843468-openshift-default-6b9fcfb55c-dc2fv"] Jul 06 20:40:12.072855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:12.072821 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" path="/var/lib/kubelet/pods/ee0d84cc-e75c-4b61-b2ae-2c629564cda1/volumes" Jul 06 20:40:15.332235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.332198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:15.433955 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.433891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:15.529590 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.529545 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:15.530411 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530386 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" containerName="istio-proxy" Jul 06 20:40:15.530555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530425 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" containerName="istio-proxy" Jul 06 20:40:15.530555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530446 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="360e8302-f87f-456f-b8fd-6e02615c8b79" containerName="maas-api" Jul 06 20:40:15.530555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530454 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="360e8302-f87f-456f-b8fd-6e02615c8b79" containerName="maas-api" Jul 06 20:40:15.530725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530616 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0d84cc-e75c-4b61-b2ae-2c629564cda1" containerName="istio-proxy" Jul 06 20:40:15.530725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.530632 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="360e8302-f87f-456f-b8fd-6e02615c8b79" containerName="maas-api" Jul 06 20:40:15.533812 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.533791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.536231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.536211 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-25f56f69-openshift-default-dockercfg-fj9gr\"" Jul 06 20:40:15.550248 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.550214 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:15.588529 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.588464 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:15.695144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695273 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnjh\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.695524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.695403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796588 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796588 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnjh\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796588 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796544 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796588 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796613 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.796792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.796669 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.797032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.797007 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.798655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.798634 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.798981 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.798962 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.806379 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.806353 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnjh\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.806695 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.806674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token\") pod \"e2e-derive-25f56f69-openshift-default-586549bb47-clhsn\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:15.848012 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:15.847965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:16.012462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.012437 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:16.013985 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:40:16.013961 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb37986_374b_496a_9e53_5853fcf6b21a.slice/crio-7fbbaaf2eff2f449f0ef63c58819c74bc12fc57085d46de553df4dff62f56b94 WatchSource:0}: Error finding container 7fbbaaf2eff2f449f0ef63c58819c74bc12fc57085d46de553df4dff62f56b94: Status 404 returned error can't find the container with id 7fbbaaf2eff2f449f0ef63c58819c74bc12fc57085d46de553df4dff62f56b94 Jul 06 20:40:16.016074 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.016044 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:40:16.016157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.016106 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:40:16.016157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.016134 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:40:16.189244 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.189208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" event={"ID":"8fb37986-374b-496a-9e53-5853fcf6b21a","Type":"ContainerStarted","Data":"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14"} Jul 06 20:40:16.189422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.189269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" event={"ID":"8fb37986-374b-496a-9e53-5853fcf6b21a","Type":"ContainerStarted","Data":"7fbbaaf2eff2f449f0ef63c58819c74bc12fc57085d46de553df4dff62f56b94"} Jul 06 20:40:16.213194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.213150 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" podStartSLOduration=1.2131337150000001 podStartE2EDuration="1.213133715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:16.210331976 +0000 UTC m=+1532.700955527" watchObservedRunningTime="2026-07-06 20:40:16.213133715 +0000 UTC m=+1532.703757266" Jul 06 20:40:16.690083 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.690051 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:16.780859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.780819 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:16.784547 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.784524 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:16.787131 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.787111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-25f56f69-serving-cert\"" Jul 06 20:40:16.791280 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.791248 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:16.797882 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.797857 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:16.848618 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.848588 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:16.891124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.891082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:16.907193 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.907159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxln\" (UniqueName: \"kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:16.907346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.907220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:16.907346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:16.907263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.007788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.007696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.007788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.007747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.008069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.007848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxln\" (UniqueName: \"kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.008069 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:40:17.007854 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-25f56f69-serving-cert: secret "maas-api-e2e-derive-25f56f69-serving-cert" not found Jul 06 20:40:17.008069 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:40:17.007957 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls podName:740011e7-34b3-4bc9-a410-d37b7799ab33 nodeName:}" failed. No retries permitted until 2026-07-06 20:40:17.507938335 +0000 UTC m=+1533.998561866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls") pod "maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" (UID: "740011e7-34b3-4bc9-a410-d37b7799ab33") : secret "maas-api-e2e-derive-25f56f69-serving-cert" not found Jul 06 20:40:17.008470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.008452 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.020644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.020619 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxln\" (UniqueName: \"kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.286993 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.286897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:17.420429 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.420395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:17.513443 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.513407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.515718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.515698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") pod \"maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.522813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.522786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:17.698128 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.698100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:17.837502 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.837477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:17.839010 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:40:17.838980 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740011e7_34b3_4bc9_a410_d37b7799ab33.slice/crio-b9e128c0a6876dadcb989d0682ed94754a20ea44a322decda084f1fd3a55cb41 WatchSource:0}: Error finding container b9e128c0a6876dadcb989d0682ed94754a20ea44a322decda084f1fd3a55cb41: Status 404 returned error can't find the container with id b9e128c0a6876dadcb989d0682ed94754a20ea44a322decda084f1fd3a55cb41 Jul 06 20:40:17.853347 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:17.853328 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:18.201699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:18.201671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" event={"ID":"740011e7-34b3-4bc9-a410-d37b7799ab33","Type":"ContainerStarted","Data":"b9e128c0a6876dadcb989d0682ed94754a20ea44a322decda084f1fd3a55cb41"} Jul 06 20:40:18.201854 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:18.201838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:18.202800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:18.202783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:19.209088 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:19.209053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" event={"ID":"740011e7-34b3-4bc9-a410-d37b7799ab33","Type":"ContainerStarted","Data":"816d77bea212fc3bb3ea68e094dd84c89a7eb3da436467d31d3a74391556e7f7"} Jul 06 20:40:19.209486 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:19.209263 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:19.230999 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:19.230945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" podStartSLOduration=2.905402587 podStartE2EDuration="3.230929703s" totalImagesPullingTime="325.527116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:40:19.229852936 +0000 UTC m=+1535.720476486" watchObservedRunningTime="2026-07-06 20:40:19.230929703 +0000 UTC m=+1535.721553253" Jul 06 20:40:20.903936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:20.903277 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:21.394620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.394586 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:21.511349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.511317 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:21.511516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.511494 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" podUID="8fb37986-374b-496a-9e53-5853fcf6b21a" containerName="istio-proxy" containerID="cri-o://6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14" gracePeriod=30 Jul 06 20:40:21.544541 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.544502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:21.668198 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.668166 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:22.000982 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.996146 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:22.000982 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:21.996417 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" podUID="740011e7-34b3-4bc9-a410-d37b7799ab33" containerName="maas-api" containerID="cri-o://816d77bea212fc3bb3ea68e094dd84c89a7eb3da436467d31d3a74391556e7f7" gracePeriod=30 Jul 06 20:40:22.008323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.008301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:22.230084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.230057 2574 generic.go:356] "Generic (PLEG): container finished" podID="740011e7-34b3-4bc9-a410-d37b7799ab33" containerID="816d77bea212fc3bb3ea68e094dd84c89a7eb3da436467d31d3a74391556e7f7" exitCode=0 Jul 06 20:40:22.230293 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.230132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" event={"ID":"740011e7-34b3-4bc9-a410-d37b7799ab33","Type":"ContainerDied","Data":"816d77bea212fc3bb3ea68e094dd84c89a7eb3da436467d31d3a74391556e7f7"} Jul 06 20:40:22.264266 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.264216 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:22.354458 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.354426 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kxln\" (UniqueName: \"kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln\") pod \"740011e7-34b3-4bc9-a410-d37b7799ab33\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " Jul 06 20:40:22.354607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.354512 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") pod \"740011e7-34b3-4bc9-a410-d37b7799ab33\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " Jul 06 20:40:22.354607 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.354546 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca\") pod \"740011e7-34b3-4bc9-a410-d37b7799ab33\" (UID: \"740011e7-34b3-4bc9-a410-d37b7799ab33\") " Jul 06 20:40:22.354923 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.354886 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "740011e7-34b3-4bc9-a410-d37b7799ab33" (UID: "740011e7-34b3-4bc9-a410-d37b7799ab33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:40:22.356449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.356425 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln" (OuterVolumeSpecName: "kube-api-access-4kxln") pod "740011e7-34b3-4bc9-a410-d37b7799ab33" (UID: "740011e7-34b3-4bc9-a410-d37b7799ab33"). InnerVolumeSpecName "kube-api-access-4kxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:22.356531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.356430 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "740011e7-34b3-4bc9-a410-d37b7799ab33" (UID: "740011e7-34b3-4bc9-a410-d37b7799ab33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:40:22.455212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.455177 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/740011e7-34b3-4bc9-a410-d37b7799ab33-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:22.455212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.455206 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/740011e7-34b3-4bc9-a410-d37b7799ab33-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:22.455212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.455218 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxln\" (UniqueName: \"kubernetes.io/projected/740011e7-34b3-4bc9-a410-d37b7799ab33-kube-api-access-4kxln\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:22.969464 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:22.969431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:23.237318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:23.237233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" event={"ID":"740011e7-34b3-4bc9-a410-d37b7799ab33","Type":"ContainerDied","Data":"b9e128c0a6876dadcb989d0682ed94754a20ea44a322decda084f1fd3a55cb41"} Jul 06 20:40:23.237318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:23.237249 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p" Jul 06 20:40:23.237318 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:23.237289 2574 scope.go:117] "RemoveContainer" containerID="816d77bea212fc3bb3ea68e094dd84c89a7eb3da436467d31d3a74391556e7f7" Jul 06 20:40:23.266603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:23.266575 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:23.273369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:23.273349 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-25f56f69-6c66dbbfb8-wb48p"] Jul 06 20:40:24.073526 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:24.073491 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740011e7-34b3-4bc9-a410-d37b7799ab33" path="/var/lib/kubelet/pods/740011e7-34b3-4bc9-a410-d37b7799ab33/volumes" Jul 06 20:40:27.172396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.172373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:27.197044 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197020 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197054 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197080 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197096 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197158 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197122 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197191 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197239 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197277 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgnjh\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert\") pod \"8fb37986-374b-496a-9e53-5853fcf6b21a\" (UID: \"8fb37986-374b-496a-9e53-5853fcf6b21a\") " Jul 06 20:40:27.197584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197272 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:27.197584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197456 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:27.197584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197480 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:27.197745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197707 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.197745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197729 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.197745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197744 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.197929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197766 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:40:27.197995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.197952 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:27.199344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.199322 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:40:27.199432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.199388 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh" (OuterVolumeSpecName: "kube-api-access-wgnjh") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "kube-api-access-wgnjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:27.199557 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.199534 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:40:27.199698 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.199668 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fb37986-374b-496a-9e53-5853fcf6b21a" (UID: "8fb37986-374b-496a-9e53-5853fcf6b21a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:40:27.262116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.262061 2574 generic.go:356] "Generic (PLEG): container finished" podID="8fb37986-374b-496a-9e53-5853fcf6b21a" containerID="6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14" exitCode=0 Jul 06 20:40:27.262206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.262111 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" Jul 06 20:40:27.262206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.262147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" event={"ID":"8fb37986-374b-496a-9e53-5853fcf6b21a","Type":"ContainerDied","Data":"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14"} Jul 06 20:40:27.262206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.262198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn" event={"ID":"8fb37986-374b-496a-9e53-5853fcf6b21a","Type":"ContainerDied","Data":"7fbbaaf2eff2f449f0ef63c58819c74bc12fc57085d46de553df4dff62f56b94"} Jul 06 20:40:27.262315 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.262217 2574 scope.go:117] "RemoveContainer" containerID="6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14" Jul 06 20:40:27.271453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.271438 2574 scope.go:117] "RemoveContainer" containerID="6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14" Jul 06 20:40:27.271672 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:40:27.271654 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14\": container with ID starting with 6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14 not found: ID does not exist" containerID="6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14" Jul 06 20:40:27.271726 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.271680 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14"} err="failed to get container status \"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14\": rpc error: code = NotFound desc = could not find container \"6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14\": container with ID starting with 6fe2c987b3cd4da3c03bc45973b90db20100eac55d4ac6b18ac45a5807185f14 not found: ID does not exist" Jul 06 20:40:27.287300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.287275 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:27.293673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.293648 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-25f56f69-openshift-default-586549bb47-clhsn"] Jul 06 20:40:27.298371 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298350 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.298480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298372 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.298480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298386 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.298480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298398 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb37986-374b-496a-9e53-5853fcf6b21a-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.298480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298410 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgnjh\" (UniqueName: \"kubernetes.io/projected/8fb37986-374b-496a-9e53-5853fcf6b21a-kube-api-access-wgnjh\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.298480 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.298422 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb37986-374b-496a-9e53-5853fcf6b21a-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:40:27.777994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.777968 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:27.877399 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:27.877363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:28.072174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:28.072099 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb37986-374b-496a-9e53-5853fcf6b21a" path="/var/lib/kubelet/pods/8fb37986-374b-496a-9e53-5853fcf6b21a/volumes" Jul 06 20:40:28.291839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:28.291808 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:28.391343 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:28.391275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:28.490936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:28.490890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:29.965481 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:29.965449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:35.679724 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:35.679681 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:36.008196 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:36.008105 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:36.108108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:36.108073 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:40:36.389884 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:40:36.389809 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:02.418030 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:02.417993 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:08.247358 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:08.247325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:08.347282 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:08.347256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:08.447099 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:08.447071 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:24.335495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:24.335459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:30.818447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:30.818416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:30.873812 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:30.873783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:40.810813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:40.810733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:41.647948 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:41.647883 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:47.890627 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:47.890597 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:47.996394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:47.996363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:48.105013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:48.104979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:48.203152 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:48.203077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:48.984584 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:48.984548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:49.090422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:49.090390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:49.182842 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:49.182810 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:50.887861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:50.887828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:56.796300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:56.796271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:56.898951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:56.898924 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:41:57.479364 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:41:57.479331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.007418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.007385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.109414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.109386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.277962 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.277872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.577555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.577461 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:04.578085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578065 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb37986-374b-496a-9e53-5853fcf6b21a" containerName="istio-proxy" Jul 06 20:42:04.578085 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb37986-374b-496a-9e53-5853fcf6b21a" containerName="istio-proxy" Jul 06 20:42:04.578251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578097 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740011e7-34b3-4bc9-a410-d37b7799ab33" containerName="maas-api" Jul 06 20:42:04.578251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578107 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="740011e7-34b3-4bc9-a410-d37b7799ab33" containerName="maas-api" Jul 06 20:42:04.578251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578215 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="740011e7-34b3-4bc9-a410-d37b7799ab33" containerName="maas-api" Jul 06 20:42:04.578251 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.578233 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb37986-374b-496a-9e53-5853fcf6b21a" containerName="istio-proxy" Jul 06 20:42:04.581572 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.581552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.583940 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.583918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4e348223-openshift-default-dockercfg-4s5kx\"" Jul 06 20:42:04.602667 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.602646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:04.606434 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.606411 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.646129 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646230 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646230 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646314 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646314 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tc8\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.646445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.646415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747203 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747466 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747466 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tc8\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747531 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747571 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.747839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.748000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747881 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.748000 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.747944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.748151 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.748132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.749561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.749542 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.749789 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.749770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.756179 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.756157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.756263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.756163 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tc8\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8\") pod \"e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:04.801991 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.801956 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:04.892176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:04.892097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:05.044801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.044771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:05.046218 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:42:05.046189 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22530c5_f595_42d4_bffe_60b6a8b40759.slice/crio-801f5900e89097a8429116c27682fa68685306c4f0e4aed14fa2be3f88cd2d1d WatchSource:0}: Error finding container 801f5900e89097a8429116c27682fa68685306c4f0e4aed14fa2be3f88cd2d1d: Status 404 returned error can't find the container with id 801f5900e89097a8429116c27682fa68685306c4f0e4aed14fa2be3f88cd2d1d Jul 06 20:42:05.048260 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.048226 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:05.048344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.048293 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:05.048344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.048321 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:05.809339 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.809303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerStarted","Data":"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79"} Jul 06 20:42:05.809501 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.809366 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerStarted","Data":"801f5900e89097a8429116c27682fa68685306c4f0e4aed14fa2be3f88cd2d1d"} Jul 06 20:42:05.831300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.831246 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" podStartSLOduration=1.831226816 podStartE2EDuration="1.831226816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:05.830509985 +0000 UTC m=+1642.321133556" watchObservedRunningTime="2026-07-06 20:42:05.831226816 +0000 UTC m=+1642.321850368" Jul 06 20:42:05.892970 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:05.892949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:06.122143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:06.122044 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:06.755446 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:06.755416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:06.897960 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:06.897937 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:07.820168 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:07.820144 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:07.821151 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:07.821131 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:17.788942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.788889 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:17.854141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.854108 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:17.857606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.857587 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:17.859812 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.859793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-4e348223-serving-cert\"" Jul 06 20:42:17.869673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.869641 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:17.898468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.898441 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:17.955289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.955254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:17.955419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.955321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qs6j\" (UniqueName: \"kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:17.955489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.955457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:17.988695 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:17.988663 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:18.056701 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.056631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.056701 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.056675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qs6j\" (UniqueName: \"kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.056882 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.056726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.057371 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.057350 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.059059 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.059038 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.064538 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.064517 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qs6j\" (UniqueName: \"kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j\") pod \"maas-api-e2e-mt-4e348223-57cf855966-vb6kg\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.094866 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.094830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:18.169116 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.169093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.305725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.305702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:18.308303 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:42:18.308238 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e7d8b1_3493_41aa_ba12_652bd33c644b.slice/crio-63d5dd00ac7e4c63b6834745db2ad1a7891e123aa295949db6dc9f1ea8353a1b WatchSource:0}: Error finding container 63d5dd00ac7e4c63b6834745db2ad1a7891e123aa295949db6dc9f1ea8353a1b: Status 404 returned error can't find the container with id 63d5dd00ac7e4c63b6834745db2ad1a7891e123aa295949db6dc9f1ea8353a1b Jul 06 20:42:18.393792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.393758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:18.494658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.494623 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:18.881077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.881041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" event={"ID":"71e7d8b1-3493-41aa-ba12-652bd33c644b","Type":"ContainerStarted","Data":"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777"} Jul 06 20:42:18.881499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.881102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" event={"ID":"71e7d8b1-3493-41aa-ba12-652bd33c644b","Type":"ContainerStarted","Data":"63d5dd00ac7e4c63b6834745db2ad1a7891e123aa295949db6dc9f1ea8353a1b"} Jul 06 20:42:18.881499 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.881203 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:18.901126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:18.901085 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" podStartSLOduration=1.461622891 podStartE2EDuration="1.901069785s" totalImagesPullingTime="439.446894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:18.898630007 +0000 UTC m=+1655.389253574" watchObservedRunningTime="2026-07-06 20:42:18.901069785 +0000 UTC m=+1655.391693339" Jul 06 20:42:22.266916 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:22.266880 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:23.000291 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:23.000219 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:23.085297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:23.085267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:23.388311 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:23.388223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:24.900997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:24.900153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:28.947508 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.947475 2574 generic.go:356] "Generic (PLEG): container finished" podID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerID="9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79" exitCode=0 Jul 06 20:42:28.947878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.947550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerDied","Data":"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79"} Jul 06 20:42:28.947991 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.947973 2574 scope.go:117] "RemoveContainer" containerID="9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79" Jul 06 20:42:28.948991 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.948967 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:28.949061 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.949043 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:28.949102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:28.949081 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:29.483062 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:29.483037 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:29.582165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:29.582139 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:29.903767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:29.903622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:29.955166 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:29.955135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerStarted","Data":"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb"} Jul 06 20:42:30.317936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.317275 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:30.317936 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.317555 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:30.318328 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.318295 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" podUID="71e7d8b1-3493-41aa-ba12-652bd33c644b" containerName="maas-api" containerID="cri-o://a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777" gracePeriod=30 Jul 06 20:42:30.411987 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.411955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:30.607727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.607645 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:30.684214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.684179 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca\") pod \"71e7d8b1-3493-41aa-ba12-652bd33c644b\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " Jul 06 20:42:30.684383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.684282 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls\") pod \"71e7d8b1-3493-41aa-ba12-652bd33c644b\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " Jul 06 20:42:30.684383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.684330 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qs6j\" (UniqueName: \"kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j\") pod \"71e7d8b1-3493-41aa-ba12-652bd33c644b\" (UID: \"71e7d8b1-3493-41aa-ba12-652bd33c644b\") " Jul 06 20:42:30.685191 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.685148 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71e7d8b1-3493-41aa-ba12-652bd33c644b" (UID: "71e7d8b1-3493-41aa-ba12-652bd33c644b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:42:30.686939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.685577 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e7d8b1-3493-41aa-ba12-652bd33c644b-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:30.692934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.689553 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71e7d8b1-3493-41aa-ba12-652bd33c644b" (UID: "71e7d8b1-3493-41aa-ba12-652bd33c644b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:42:30.692934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.691282 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j" (OuterVolumeSpecName: "kube-api-access-6qs6j") pod "71e7d8b1-3493-41aa-ba12-652bd33c644b" (UID: "71e7d8b1-3493-41aa-ba12-652bd33c644b"). InnerVolumeSpecName "kube-api-access-6qs6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:42:30.786750 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.786713 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e7d8b1-3493-41aa-ba12-652bd33c644b-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:30.786917 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.786756 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qs6j\" (UniqueName: \"kubernetes.io/projected/71e7d8b1-3493-41aa-ba12-652bd33c644b-kube-api-access-6qs6j\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:30.893684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.893220 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:30.893684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.893339 2574 patch_prober.go:28] interesting pod/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 06 20:42:30.893684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.893382 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 06 20:42:30.962392 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.962366 2574 generic.go:356] "Generic (PLEG): container finished" podID="71e7d8b1-3493-41aa-ba12-652bd33c644b" containerID="a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777" exitCode=0 Jul 06 20:42:30.962722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.962396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" event={"ID":"71e7d8b1-3493-41aa-ba12-652bd33c644b","Type":"ContainerDied","Data":"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777"} Jul 06 20:42:30.962722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.962420 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" Jul 06 20:42:30.962722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.962450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg" event={"ID":"71e7d8b1-3493-41aa-ba12-652bd33c644b","Type":"ContainerDied","Data":"63d5dd00ac7e4c63b6834745db2ad1a7891e123aa295949db6dc9f1ea8353a1b"} Jul 06 20:42:30.962722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.962474 2574 scope.go:117] "RemoveContainer" containerID="a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777" Jul 06 20:42:30.972406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.972390 2574 scope.go:117] "RemoveContainer" containerID="a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777" Jul 06 20:42:30.972632 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:42:30.972616 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777\": container with ID starting with a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777 not found: ID does not exist" containerID="a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777" Jul 06 20:42:30.972692 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.972638 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777"} err="failed to get container status \"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777\": rpc error: code = NotFound desc = could not find container \"a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777\": container with ID starting with a1013d4b4fb4d6b428e09cb9abc60e0fa18763392704ae828274000cb99fc777 not found: ID does not exist" Jul 06 20:42:30.987548 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.987521 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:30.993101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:30.993072 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-4e348223-57cf855966-vb6kg"] Jul 06 20:42:31.893230 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:31.893193 2574 patch_prober.go:28] interesting pod/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 06 20:42:31.893519 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:31.893486 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 06 20:42:32.072125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:32.072098 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e7d8b1-3493-41aa-ba12-652bd33c644b" path="/var/lib/kubelet/pods/71e7d8b1-3493-41aa-ba12-652bd33c644b/volumes" Jul 06 20:42:32.893019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:32.892975 2574 patch_prober.go:28] interesting pod/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 06 20:42:32.893220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:32.893059 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 06 20:42:33.896126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:33.896098 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:33.896593 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:33.896574 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:33.897220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:33.897196 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:36.418622 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.418595 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:36.515342 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.515314 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:36.667295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.667257 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:36.667492 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.667465 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" containerID="cri-o://33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb" gracePeriod=30 Jul 06 20:42:36.682365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.682305 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:36.904495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:36.904460 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:37.028800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:37.028728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:37.125125 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:37.125098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:37.245377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:37.245348 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:41.086510 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:41.086475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:41.157523 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:41.157492 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:41.310879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:41.310842 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:41.410809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:41.410772 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:41.923372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:41.923350 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:42.031153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.031080 2574 generic.go:356] "Generic (PLEG): container finished" podID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerID="33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb" exitCode=0 Jul 06 20:42:42.031153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.031128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerDied","Data":"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb"} Jul 06 20:42:42.031316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.031167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" event={"ID":"c22530c5-f595-42d4-bffe-60b6a8b40759","Type":"ContainerDied","Data":"801f5900e89097a8429116c27682fa68685306c4f0e4aed14fa2be3f88cd2d1d"} Jul 06 20:42:42.031316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.031188 2574 scope.go:117] "RemoveContainer" containerID="33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb" Jul 06 20:42:42.031316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.031129 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b" Jul 06 20:42:42.039873 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.039858 2574 scope.go:117] "RemoveContainer" containerID="9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79" Jul 06 20:42:42.047890 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.047870 2574 scope.go:117] "RemoveContainer" containerID="33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb" Jul 06 20:42:42.048170 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:42:42.048153 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb\": container with ID starting with 33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb not found: ID does not exist" containerID="33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb" Jul 06 20:42:42.048221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.048178 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb"} err="failed to get container status \"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb\": rpc error: code = NotFound desc = could not find container \"33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb\": container with ID starting with 33f5aaa2c5a5b5dc1f7f2d04375168f1fd888ed83a9e34606d86840d63ee99eb not found: ID does not exist" Jul 06 20:42:42.048221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.048194 2574 scope.go:117] "RemoveContainer" containerID="9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79" Jul 06 20:42:42.048421 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:42:42.048405 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79\": container with ID starting with 9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79 not found: ID does not exist" containerID="9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79" Jul 06 20:42:42.048469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.048426 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79"} err="failed to get container status \"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79\": rpc error: code = NotFound desc = could not find container \"9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79\": container with ID starting with 9dba5418ea245c98c0fde0188021dc85e06ab68d2edea605ccf65241c2c6eb79 not found: ID does not exist" Jul 06 20:42:42.088722 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088703 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088729 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088791 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088818 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088883 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088933 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088972 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.088999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54tc8\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089067 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089032 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket\") pod \"c22530c5-f595-42d4-bffe-60b6a8b40759\" (UID: \"c22530c5-f595-42d4-bffe-60b6a8b40759\") " Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089089 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089186 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089213 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089383 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089450 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089470 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.089496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.089484 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22530c5-f595-42d4-bffe-60b6a8b40759-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.090045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.090022 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data" (OuterVolumeSpecName: "istio-data") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:42:42.091117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.091091 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:42:42.091221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.091147 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token" (OuterVolumeSpecName: "istio-token") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:42:42.091439 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.091417 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8" (OuterVolumeSpecName: "kube-api-access-54tc8") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "kube-api-access-54tc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:42:42.091439 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.091420 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c22530c5-f595-42d4-bffe-60b6a8b40759" (UID: "c22530c5-f595-42d4-bffe-60b6a8b40759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:42:42.190641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190615 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.190641 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190637 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.190783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190647 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.190783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190655 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tc8\" (UniqueName: \"kubernetes.io/projected/c22530c5-f595-42d4-bffe-60b6a8b40759-kube-api-access-54tc8\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.190783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190663 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.190783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.190671 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22530c5-f595-42d4-bffe-60b6a8b40759-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:42:42.356108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.356081 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:42.362994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:42.362970 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4e348223-openshift-default-67884f57d7-w2v7b"] Jul 06 20:42:43.270815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:43.270781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:43.382612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:43.382547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:43.512018 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:43.511990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:43.652533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:43.652440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:44.073117 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:44.073087 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" path="/var/lib/kubelet/pods/c22530c5-f595-42d4-bffe-60b6a8b40759/volumes" Jul 06 20:42:49.373264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:49.373210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:49.471270 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:49.471245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:56.152397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.152367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:56.499704 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.499668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:42:56.500270 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500254 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7d8b1-3493-41aa-ba12-652bd33c644b" containerName="maas-api" Jul 06 20:42:56.500327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500274 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7d8b1-3493-41aa-ba12-652bd33c644b" containerName="maas-api" Jul 06 20:42:56.500327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500303 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:56.500327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500311 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:56.500327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500322 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:56.500454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500331 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:56.500454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500426 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:56.500454 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.500439 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7d8b1-3493-41aa-ba12-652bd33c644b" containerName="maas-api" Jul 06 20:42:56.503856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.503831 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.508942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.508921 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bd79a608-openshift-default-dockercfg-dpmkw\"" Jul 06 20:42:56.521793 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.521763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:42:56.550259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.550227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:56.603358 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv777\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.603792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.603715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704320 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv777\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.704797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704671 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.705025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.704824 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.705103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.705078 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.705165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.705111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.705234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.705201 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.706837 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.706801 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.707975 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.707933 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.714594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.714568 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:56.714993 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.714976 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.715093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.715001 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv777\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777\") pod \"e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.814452 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.814362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:56.986747 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.986722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:42:56.988550 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:42:56.988513 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263196d8_cf35_4de6_a6ed_bf02129bb30e.slice/crio-d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501 WatchSource:0}: Error finding container d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501: Status 404 returned error can't find the container with id d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501 Jul 06 20:42:56.990749 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.990720 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:56.990818 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.990781 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:56.990818 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:56.990812 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:42:57.132237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:57.131310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerStarted","Data":"c768d70bc7b80a24638b71a432cc478c5599e796dbc48604cd5b9440e7e1f73e"} Jul 06 20:42:57.132237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:57.131372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerStarted","Data":"d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501"} Jul 06 20:42:57.814441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:57.814413 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:58.718983 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:58.718951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:58.820006 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:58.819978 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:58.915208 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:58.915155 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podStartSLOduration=2.915135835 podStartE2EDuration="2.915135835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:42:57.162844293 +0000 UTC m=+1693.653467843" watchObservedRunningTime="2026-07-06 20:42:58.915135835 +0000 UTC m=+1695.405759457" Jul 06 20:42:58.925941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:58.925890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:59.143284 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.143182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:59.149496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.149471 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:42:59.279100 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.279034 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:59.444101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.444072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:59.487942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.487916 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:42:59.488349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.488337 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22530c5-f595-42d4-bffe-60b6a8b40759" containerName="istio-proxy" Jul 06 20:42:59.490421 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.490404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.498206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.498185 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-bd79a608-serving-cert\"" Jul 06 20:42:59.520353 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.520331 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:42:59.531914 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.531870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.532008 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.531937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzpg\" (UniqueName: \"kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.532054 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.532019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.555662 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.555634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:42:59.632808 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.632784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.632997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.632840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.632997 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.632862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzpg\" (UniqueName: \"kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.632997 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:42:59.632959 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-bd79a608-serving-cert: secret "maas-api-e2e-mt-bd79a608-serving-cert" not found Jul 06 20:42:59.633180 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:42:59.633019 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls podName:96a2106e-b888-4509-8b92-6c2d063b5128 nodeName:}" failed. No retries permitted until 2026-07-06 20:43:00.133002309 +0000 UTC m=+1696.623625836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls") pod "maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" (UID: "96a2106e-b888-4509-8b92-6c2d063b5128") : secret "maas-api-e2e-mt-bd79a608-serving-cert" not found Jul 06 20:42:59.633501 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.633478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:42:59.657384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:42:59.657360 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzpg\" (UniqueName: \"kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:00.060532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.060498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:00.136693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.136655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:00.139184 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.139154 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") pod \"maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:00.287126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.287089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:00.401278 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.401201 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:00.445049 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.445003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:00.568937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:00.568885 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:43:00.570868 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:43:00.570839 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a2106e_b888_4509_8b92_6c2d063b5128.slice/crio-3bd69fd02945c15dc1970066bf2a87e6febe143f57e766cca6fe47b56b1a6615 WatchSource:0}: Error finding container 3bd69fd02945c15dc1970066bf2a87e6febe143f57e766cca6fe47b56b1a6615: Status 404 returned error can't find the container with id 3bd69fd02945c15dc1970066bf2a87e6febe143f57e766cca6fe47b56b1a6615 Jul 06 20:43:01.155141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:01.155112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" event={"ID":"96a2106e-b888-4509-8b92-6c2d063b5128","Type":"ContainerStarted","Data":"3bd69fd02945c15dc1970066bf2a87e6febe143f57e766cca6fe47b56b1a6615"} Jul 06 20:43:02.161746 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:02.161712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" event={"ID":"96a2106e-b888-4509-8b92-6c2d063b5128","Type":"ContainerStarted","Data":"1559ccaf73d0d7cb42545fa577f1c33d53f70320ac9876f9e96615bd8be7db07"} Jul 06 20:43:02.162167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:02.161827 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:02.184734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:02.184689 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" podStartSLOduration=2.633850163 podStartE2EDuration="3.184675936s" totalImagesPullingTime="550.825773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:43:02.18227565 +0000 UTC m=+1698.672899197" watchObservedRunningTime="2026-07-06 20:43:02.184675936 +0000 UTC m=+1698.675299486" Jul 06 20:43:03.699471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:03.699436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:08.173063 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:08.172983 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:43:13.235638 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.235605 2574 generic.go:356] "Generic (PLEG): container finished" podID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerID="c768d70bc7b80a24638b71a432cc478c5599e796dbc48604cd5b9440e7e1f73e" exitCode=0 Jul 06 20:43:13.236185 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.235681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerDied","Data":"c768d70bc7b80a24638b71a432cc478c5599e796dbc48604cd5b9440e7e1f73e"} Jul 06 20:43:13.236261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.236184 2574 scope.go:117] "RemoveContainer" containerID="c768d70bc7b80a24638b71a432cc478c5599e796dbc48604cd5b9440e7e1f73e" Jul 06 20:43:13.237173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.237145 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:43:13.237281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.237248 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:43:13.237355 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:13.237298 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:43:14.142579 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:14.142540 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:43:14.243497 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:14.243463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerStarted","Data":"36adce538b89fed29f5649df3a33fe1a5a3d062696a3c798945637e92d066560"} Jul 06 20:43:14.815597 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:14.815505 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:43:14.816445 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:14.816420 2574 patch_prober.go:28] interesting pod/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 06 20:43:14.816517 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:14.816497 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 06 20:43:15.815162 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:15.815111 2574 patch_prober.go:28] interesting pod/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 06 20:43:15.815634 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:15.815186 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 06 20:43:16.815566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:16.815521 2574 patch_prober.go:28] interesting pod/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 06 20:43:16.816003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:16.815587 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 06 20:43:17.815417 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:17.815383 2574 patch_prober.go:28] interesting pod/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 06 20:43:17.815577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:17.815442 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 06 20:43:18.818956 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:18.818930 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:43:18.819308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:18.819195 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:43:18.819758 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:18.819742 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:43:28.818516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:28.818470 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:31.549033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:31.548997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:43:31.656082 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:43:31.656054 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:01.933372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:01.933332 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:02.358718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.358639 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:44:02.358861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.358820 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" podUID="96a2106e-b888-4509-8b92-6c2d063b5128" containerName="maas-api" containerID="cri-o://1559ccaf73d0d7cb42545fa577f1c33d53f70320ac9876f9e96615bd8be7db07" gracePeriod=30 Jul 06 20:44:02.416012 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.415977 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:02.504284 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.504254 2574 generic.go:356] "Generic (PLEG): container finished" podID="96a2106e-b888-4509-8b92-6c2d063b5128" containerID="1559ccaf73d0d7cb42545fa577f1c33d53f70320ac9876f9e96615bd8be7db07" exitCode=0 Jul 06 20:44:02.504425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.504322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" event={"ID":"96a2106e-b888-4509-8b92-6c2d063b5128","Type":"ContainerDied","Data":"1559ccaf73d0d7cb42545fa577f1c33d53f70320ac9876f9e96615bd8be7db07"} Jul 06 20:44:02.615642 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.615589 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:44:02.682077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.682047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca\") pod \"96a2106e-b888-4509-8b92-6c2d063b5128\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " Jul 06 20:44:02.682195 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.682130 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") pod \"96a2106e-b888-4509-8b92-6c2d063b5128\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " Jul 06 20:44:02.682195 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.682186 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvzpg\" (UniqueName: \"kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg\") pod \"96a2106e-b888-4509-8b92-6c2d063b5128\" (UID: \"96a2106e-b888-4509-8b92-6c2d063b5128\") " Jul 06 20:44:02.682353 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.682331 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "96a2106e-b888-4509-8b92-6c2d063b5128" (UID: "96a2106e-b888-4509-8b92-6c2d063b5128"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:44:02.682519 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.682503 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a2106e-b888-4509-8b92-6c2d063b5128-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:44:02.684084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.684064 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "96a2106e-b888-4509-8b92-6c2d063b5128" (UID: "96a2106e-b888-4509-8b92-6c2d063b5128"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:44:02.684183 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.684143 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg" (OuterVolumeSpecName: "kube-api-access-qvzpg") pod "96a2106e-b888-4509-8b92-6c2d063b5128" (UID: "96a2106e-b888-4509-8b92-6c2d063b5128"). InnerVolumeSpecName "kube-api-access-qvzpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:44:02.783934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.783892 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96a2106e-b888-4509-8b92-6c2d063b5128-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:44:02.783934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:02.783933 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzpg\" (UniqueName: \"kubernetes.io/projected/96a2106e-b888-4509-8b92-6c2d063b5128-kube-api-access-qvzpg\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:44:03.511868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:03.511838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" event={"ID":"96a2106e-b888-4509-8b92-6c2d063b5128","Type":"ContainerDied","Data":"3bd69fd02945c15dc1970066bf2a87e6febe143f57e766cca6fe47b56b1a6615"} Jul 06 20:44:03.512323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:03.511846 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj" Jul 06 20:44:03.512323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:03.511919 2574 scope.go:117] "RemoveContainer" containerID="1559ccaf73d0d7cb42545fa577f1c33d53f70320ac9876f9e96615bd8be7db07" Jul 06 20:44:03.537064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:03.537021 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:44:03.539744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:03.539722 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-bd79a608-798d6cd746-6b7sj"] Jul 06 20:44:04.073130 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:04.073101 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a2106e-b888-4509-8b92-6c2d063b5128" path="/var/lib/kubelet/pods/96a2106e-b888-4509-8b92-6c2d063b5128/volumes" Jul 06 20:44:08.119323 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:08.119284 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:13.170509 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:13.170459 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:44.088468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:44.088437 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:44:44.097520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:44.097498 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:44:50.231801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:50.231759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:51.551921 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:51.551878 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:44:56.611477 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:44:56.611428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:19.438083 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.438048 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:19.535854 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.535827 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:19.699463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.699391 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:45:19.699612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.699588 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" containerID="cri-o://36adce538b89fed29f5649df3a33fe1a5a3d062696a3c798945637e92d066560" gracePeriod=30 Jul 06 20:45:19.722404 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.722358 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:19.958650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:19.958571 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:20.437227 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:20.437196 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:21.025315 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:21.025258 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:21.117160 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:21.117116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:21.179477 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:21.179433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:24.950930 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.950873 2574 generic.go:356] "Generic (PLEG): container finished" podID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerID="36adce538b89fed29f5649df3a33fe1a5a3d062696a3c798945637e92d066560" exitCode=0 Jul 06 20:45:24.951256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.950925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerDied","Data":"36adce538b89fed29f5649df3a33fe1a5a3d062696a3c798945637e92d066560"} Jul 06 20:45:24.951256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.950990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" event={"ID":"263196d8-cf35-4de6-a6ed-bf02129bb30e","Type":"ContainerDied","Data":"d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501"} Jul 06 20:45:24.951256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.951011 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e773f15e31a1123ce7d3c19b04e96e024c41ddd570997cc3312d17c7344501" Jul 06 20:45:24.951256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.951026 2574 scope.go:117] "RemoveContainer" containerID="c768d70bc7b80a24638b71a432cc478c5599e796dbc48604cd5b9440e7e1f73e" Jul 06 20:45:24.956853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:24.956828 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047703 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047762 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv777\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047827 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047978 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.047989 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048085 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048122 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048191 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket\") pod \"263196d8-cf35-4de6-a6ed-bf02129bb30e\" (UID: \"263196d8-cf35-4de6-a6ed-bf02129bb30e\") " Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048508 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048634 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048715 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048884 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data" (OuterVolumeSpecName: "istio-data") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:45:25.048929 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.048880 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:45:25.051066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.051014 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:45:25.051221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.051094 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777" (OuterVolumeSpecName: "kube-api-access-nv777") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "kube-api-access-nv777". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:45:25.051440 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.051409 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token" (OuterVolumeSpecName: "istio-token") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:45:25.052089 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.052048 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "263196d8-cf35-4de6-a6ed-bf02129bb30e" (UID: "263196d8-cf35-4de6-a6ed-bf02129bb30e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:45:25.149879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149835 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.149879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149865 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/263196d8-cf35-4de6-a6ed-bf02129bb30e-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.149879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149874 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.149879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149884 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.150165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149893 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv777\" (UniqueName: \"kubernetes.io/projected/263196d8-cf35-4de6-a6ed-bf02129bb30e-kube-api-access-nv777\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.150165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149924 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.150165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.149936 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.150165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.150021 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/263196d8-cf35-4de6-a6ed-bf02129bb30e-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:45:25.462302 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.462264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:25.958168 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.958136 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd" Jul 06 20:45:25.993740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.993712 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:45:25.997601 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:25.997573 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bd79a608-openshift-default-5c5674db7-l8bcd"] Jul 06 20:45:26.072228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:26.072192 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" path="/var/lib/kubelet/pods/263196d8-cf35-4de6-a6ed-bf02129bb30e/volumes" Jul 06 20:45:29.071738 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:29.071705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:29.171449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:29.171407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:29.271296 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:29.271254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:30.644045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:30.644003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:36.034396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:36.034354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:36.752650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:36.752605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:36.854534 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:36.854486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:36.952106 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:36.952078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:37.505419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:37.505384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:43.562861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:43.562813 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:43.668122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:43.668089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:43.762951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:43.762872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:43.878375 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:43.878265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.055016 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.054979 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.157408 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.157366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.316412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.316370 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.509462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.509359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.611758 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.611725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.673032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673006 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:45:44.673375 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673363 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a2106e-b888-4509-8b92-6c2d063b5128" containerName="maas-api" Jul 06 20:45:44.673418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673377 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a2106e-b888-4509-8b92-6c2d063b5128" containerName="maas-api" Jul 06 20:45:44.673418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:44.673418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673393 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:44.673516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673469 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:44.673516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673477 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a2106e-b888-4509-8b92-6c2d063b5128" containerName="maas-api" Jul 06 20:45:44.673516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.673485 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:44.678071 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.678048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.689087 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.689059 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:45:44.713762 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.713736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.815348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815311 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:44.815348 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ds6\" (UniqueName: \"kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.815587 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.815555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.916751 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.916893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ds6\" (UniqueName: \"kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.916893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.916893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.916893 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917165 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.916964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.917630 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.917684 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.917684 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.917963 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.917945 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.919132 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.919104 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.919213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.919197 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.924994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.924977 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ds6\" (UniqueName: \"kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6\") pod \"console-679564ffdc-9dmqq\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:44.989417 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:44.989389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:45.129962 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:45.129935 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:45:45.131000 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:45:45.130979 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2da07fe_0e11_4918_bcb5_ceee97115e2e.slice/crio-65858c50db6b446ff5682a7e93286628cb7a15a9e8ce8715bc7e4730c0af4858 WatchSource:0}: Error finding container 65858c50db6b446ff5682a7e93286628cb7a15a9e8ce8715bc7e4730c0af4858: Status 404 returned error can't find the container with id 65858c50db6b446ff5682a7e93286628cb7a15a9e8ce8715bc7e4730c0af4858 Jul 06 20:45:45.363161 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:45.363072 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:45.420869 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:45.420842 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:45.778595 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:45.778557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:45.873580 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:45.873553 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:46.076312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:46.076235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679564ffdc-9dmqq" event={"ID":"d2da07fe-0e11-4918-bcb5-ceee97115e2e","Type":"ContainerStarted","Data":"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5"} Jul 06 20:45:46.076312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:46.076286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679564ffdc-9dmqq" event={"ID":"d2da07fe-0e11-4918-bcb5-ceee97115e2e","Type":"ContainerStarted","Data":"65858c50db6b446ff5682a7e93286628cb7a15a9e8ce8715bc7e4730c0af4858"} Jul 06 20:45:46.099192 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:46.099145 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679564ffdc-9dmqq" podStartSLOduration=2.099130229 podStartE2EDuration="2.099130229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:46.097609237 +0000 UTC m=+1862.588232788" watchObservedRunningTime="2026-07-06 20:45:46.099130229 +0000 UTC m=+1862.589753778" Jul 06 20:45:47.825631 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:47.825599 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:47.927163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:47.927133 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:48.525263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.525227 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:48.643437 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.643403 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:48.778329 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.778245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:48.810168 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.810141 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:45:48.842307 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.842259 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d96c4dbcd-9rjdj"] Jul 06 20:45:48.843791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.843497 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:48.843791 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.843521 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="263196d8-cf35-4de6-a6ed-bf02129bb30e" containerName="istio-proxy" Jul 06 20:45:48.846561 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.846541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.846998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst25\" (UniqueName: \"kubernetes.io/projected/069ec0b6-c77f-407e-800a-5722cdaa6fd3-kube-api-access-sst25\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847092 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-service-ca\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847219 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-trusted-ca-bundle\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-oauth-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847290 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-oauth-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.847328 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.847291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.869142 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.869120 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d96c4dbcd-9rjdj"] Jul 06 20:45:48.927448 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.927426 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:48.948237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-trusted-ca-bundle\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-oauth-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-oauth-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948350 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst25\" (UniqueName: \"kubernetes.io/projected/069ec0b6-c77f-407e-800a-5722cdaa6fd3-kube-api-access-sst25\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.948585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.948430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-service-ca\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.949037 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.949012 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-oauth-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.949228 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.949203 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-service-ca\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.949312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.949231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-trusted-ca-bundle\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.949312 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.949109 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.950803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.950784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-oauth-config\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.950944 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.950929 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/069ec0b6-c77f-407e-800a-5722cdaa6fd3-console-serving-cert\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:48.960509 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:48.960484 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst25\" (UniqueName: \"kubernetes.io/projected/069ec0b6-c77f-407e-800a-5722cdaa6fd3-kube-api-access-sst25\") pod \"console-6d96c4dbcd-9rjdj\" (UID: \"069ec0b6-c77f-407e-800a-5722cdaa6fd3\") " pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:49.039736 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.039665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.141879 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.141846 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.156045 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.156024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:49.303392 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.303369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d96c4dbcd-9rjdj"] Jul 06 20:45:49.305507 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:45:49.305479 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069ec0b6_c77f_407e_800a_5722cdaa6fd3.slice/crio-61c1f947187eee4fb83e91ffcbb37a596c702446a5bde0a8f52b8788279f8094 WatchSource:0}: Error finding container 61c1f947187eee4fb83e91ffcbb37a596c702446a5bde0a8f52b8788279f8094: Status 404 returned error can't find the container with id 61c1f947187eee4fb83e91ffcbb37a596c702446a5bde0a8f52b8788279f8094 Jul 06 20:45:49.307057 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.307033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.371681 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.371648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.642246 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.642169 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:49.813393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:49.813363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:50.008283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:50.008254 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:50.100483 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:50.100451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d96c4dbcd-9rjdj" event={"ID":"069ec0b6-c77f-407e-800a-5722cdaa6fd3","Type":"ContainerStarted","Data":"86f282ad3fcbd7e82885200080f3e32b12373df217bfb1bcb30a23a56eacdf24"} Jul 06 20:45:50.100656 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:50.100495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d96c4dbcd-9rjdj" event={"ID":"069ec0b6-c77f-407e-800a-5722cdaa6fd3","Type":"ContainerStarted","Data":"61c1f947187eee4fb83e91ffcbb37a596c702446a5bde0a8f52b8788279f8094"} Jul 06 20:45:50.107488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:50.107462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:50.128736 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:50.128696 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d96c4dbcd-9rjdj" podStartSLOduration=2.128681898 podStartE2EDuration="2.128681898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:45:50.126725255 +0000 UTC m=+1866.617348817" watchObservedRunningTime="2026-07-06 20:45:50.128681898 +0000 UTC m=+1866.619305448" Jul 06 20:45:54.297884 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:54.297855 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:54.647608 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:54.647535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:45:54.989560 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:54.989527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:45:59.156790 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:59.156757 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:59.157183 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:59.156806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:45:59.161282 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:45:59.161260 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:46:00.156207 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:00.156175 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d96c4dbcd-9rjdj" Jul 06 20:46:00.225928 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:00.225879 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:46:00.369738 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:00.369704 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:00.467989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:00.467875 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:00.537449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:00.537421 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:13.834077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:13.833971 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-679564ffdc-9dmqq" podUID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" containerName="console" containerID="cri-o://2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5" gracePeriod=15 Jul 06 20:46:14.087591 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.087533 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:46:14.152103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152072 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6ds6\" (UniqueName: \"kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152236 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152236 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152169 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152236 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152200 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152236 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152237 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152263 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152463 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152282 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle\") pod \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\" (UID: \"d2da07fe-0e11-4918-bcb5-ceee97115e2e\") " Jul 06 20:46:14.152632 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152602 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config" (OuterVolumeSpecName: "console-config") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:14.152726 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152611 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:14.152842 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152728 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:14.152889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.152857 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:14.154395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.154371 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:14.154686 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.154663 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:14.154686 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.154681 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6" (OuterVolumeSpecName: "kube-api-access-x6ds6") pod "d2da07fe-0e11-4918-bcb5-ceee97115e2e" (UID: "d2da07fe-0e11-4918-bcb5-ceee97115e2e"). InnerVolumeSpecName "kube-api-access-x6ds6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:46:14.231637 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.231611 2574 generic.go:356] "Generic (PLEG): container finished" podID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" containerID="2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5" exitCode=0 Jul 06 20:46:14.231744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.231656 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679564ffdc-9dmqq" Jul 06 20:46:14.231744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.231666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679564ffdc-9dmqq" event={"ID":"d2da07fe-0e11-4918-bcb5-ceee97115e2e","Type":"ContainerDied","Data":"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5"} Jul 06 20:46:14.231744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.231717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679564ffdc-9dmqq" event={"ID":"d2da07fe-0e11-4918-bcb5-ceee97115e2e","Type":"ContainerDied","Data":"65858c50db6b446ff5682a7e93286628cb7a15a9e8ce8715bc7e4730c0af4858"} Jul 06 20:46:14.231744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.231743 2574 scope.go:117] "RemoveContainer" containerID="2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5" Jul 06 20:46:14.240978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.240959 2574 scope.go:117] "RemoveContainer" containerID="2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5" Jul 06 20:46:14.241240 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:46:14.241221 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5\": container with ID starting with 2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5 not found: ID does not exist" containerID="2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5" Jul 06 20:46:14.241295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.241248 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5"} err="failed to get container status \"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5\": rpc error: code = NotFound desc = could not find container \"2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5\": container with ID starting with 2bd5911208602b9d9c52bed7ca43b9050ab1403f8e522e5c2fe0fbd84b8430f5 not found: ID does not exist" Jul 06 20:46:14.253056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253032 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253056 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253071 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253084 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ds6\" (UniqueName: \"kubernetes.io/projected/d2da07fe-0e11-4918-bcb5-ceee97115e2e-kube-api-access-x6ds6\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253097 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253109 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2da07fe-0e11-4918-bcb5-ceee97115e2e-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253190 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253122 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2da07fe-0e11-4918-bcb5-ceee97115e2e-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:14.253603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.253585 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:46:14.257484 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:14.257459 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679564ffdc-9dmqq"] Jul 06 20:46:15.647339 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:15.647307 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:16.071715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:16.071686 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" path="/var/lib/kubelet/pods/d2da07fe-0e11-4918-bcb5-ceee97115e2e/volumes" Jul 06 20:46:21.459425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:21.459390 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:21.547470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:21.547443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:21.831855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:21.831782 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:22.129167 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.129093 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:46:22.129651 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.129635 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" containerName="console" Jul 06 20:46:22.129702 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.129654 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" containerName="console" Jul 06 20:46:22.129766 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.129755 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2da07fe-0e11-4918-bcb5-ceee97115e2e" containerName="console" Jul 06 20:46:22.133262 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.133242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.136767 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.136752 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-37baf2-openshift-default-dockercfg-prxtl\"" Jul 06 20:46:22.145346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.145322 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:46:22.156503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.156481 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:22.213026 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213325 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mb9\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.213325 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.213297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314185 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314185 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mb9\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314414 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314704 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314811 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314787 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.314988 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.314963 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.315111 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.315003 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.315111 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.315017 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.315111 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.315082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.316689 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.316669 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.316890 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.316872 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.323795 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.323768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.323982 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.323963 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mb9\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9\") pod \"e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.443941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.443920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:22.519580 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.519550 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:22.587448 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.587423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:46:22.589936 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:46:22.589888 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587583a4_b03d_4418_a3ec_326b872ce9ab.slice/crio-8c03b3be76c879ed0f3ac7d2bbcb0782faf96290cfba9d6cb83aec814cdf5430 WatchSource:0}: Error finding container 8c03b3be76c879ed0f3ac7d2bbcb0782faf96290cfba9d6cb83aec814cdf5430: Status 404 returned error can't find the container with id 8c03b3be76c879ed0f3ac7d2bbcb0782faf96290cfba9d6cb83aec814cdf5430 Jul 06 20:46:22.592550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.592509 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:22.592659 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.592593 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:22.592659 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:22.592632 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:23.281310 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:23.281272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerStarted","Data":"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1"} Jul 06 20:46:23.281494 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:23.281330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerStarted","Data":"8c03b3be76c879ed0f3ac7d2bbcb0782faf96290cfba9d6cb83aec814cdf5430"} Jul 06 20:46:23.306500 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:23.306455 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podStartSLOduration=1.30644082 podStartE2EDuration="1.30644082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:23.304360881 +0000 UTC m=+1899.794984442" watchObservedRunningTime="2026-07-06 20:46:23.30644082 +0000 UTC m=+1899.797064369" Jul 06 20:46:23.444513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:23.444484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:23.449344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:23.449322 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:24.286798 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:24.286770 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:24.287735 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:24.287715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:25.246793 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.246751 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6859dcd84b-wzf49" podUID="9f10f493-9c24-4c41-9a90-4e897969208e" containerName="console" containerID="cri-o://06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0" gracePeriod=15 Jul 06 20:46:25.296640 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.296611 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:25.395733 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.395700 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:25.508786 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.508737 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:46:25.555036 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.554997 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:46:25.555564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.555545 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f10f493-9c24-4c41-9a90-4e897969208e" containerName="console" Jul 06 20:46:25.555564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.555564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f10f493-9c24-4c41-9a90-4e897969208e" containerName="console" Jul 06 20:46:25.555707 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.555644 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f10f493-9c24-4c41-9a90-4e897969208e" containerName="console" Jul 06 20:46:25.559512 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.559492 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.562719 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.562696 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-37baf2-serving-cert\"" Jul 06 20:46:25.573801 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.573775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:46:25.605013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.604990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:25.645084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645058 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzvc2\" (UniqueName: \"kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645196 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645129 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645201 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645261 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645235 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645280 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config\") pod \"9f10f493-9c24-4c41-9a90-4e897969208e\" (UID: \"9f10f493-9c24-4c41-9a90-4e897969208e\") " Jul 06 20:46:25.645521 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.645710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.645710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645614 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:25.645710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645628 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:25.645710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645677 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:25.645984 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.645984 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.645940 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config" (OuterVolumeSpecName: "console-config") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:46:25.646070 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.646034 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-trusted-ca-bundle\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.646070 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.646053 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.646135 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.646068 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-oauth-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.646135 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.646084 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f10f493-9c24-4c41-9a90-4e897969208e-console-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.647264 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.647241 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2" (OuterVolumeSpecName: "kube-api-access-qzvc2") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "kube-api-access-qzvc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:46:25.647369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.647349 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:25.647369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.647362 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f10f493-9c24-4c41-9a90-4e897969208e" (UID: "9f10f493-9c24-4c41-9a90-4e897969208e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:46:25.747471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.747636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.747636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.747636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747557 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-serving-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.747636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747567 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvc2\" (UniqueName: \"kubernetes.io/projected/9f10f493-9c24-4c41-9a90-4e897969208e-kube-api-access-qzvc2\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.747636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.747578 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f10f493-9c24-4c41-9a90-4e897969208e-console-oauth-config\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:46:25.747817 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:46:25.747654 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-37baf2-serving-cert: secret "maas-api-e2e-inf-a-37baf2-serving-cert" not found Jul 06 20:46:25.747817 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:46:25.747716 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls podName:4dec807b-15ae-425c-9efd-967d7914b8b0 nodeName:}" failed. No retries permitted until 2026-07-06 20:46:26.24770035 +0000 UTC m=+1902.738323881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls") pod "maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" (UID: "4dec807b-15ae-425c-9efd-967d7914b8b0") : secret "maas-api-e2e-inf-a-37baf2-serving-cert" not found Jul 06 20:46:25.748100 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.748082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.756405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.756384 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:25.913549 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:25.913502 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:26.109321 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.109292 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:26.208287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.208210 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:26.251388 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.251355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:26.253876 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.253851 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") pod \"maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:26.300708 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.300677 2574 generic.go:356] "Generic (PLEG): container finished" podID="9f10f493-9c24-4c41-9a90-4e897969208e" containerID="06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0" exitCode=0 Jul 06 20:46:26.301105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.300723 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6859dcd84b-wzf49" Jul 06 20:46:26.301105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.300759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6859dcd84b-wzf49" event={"ID":"9f10f493-9c24-4c41-9a90-4e897969208e","Type":"ContainerDied","Data":"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0"} Jul 06 20:46:26.301105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.300805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6859dcd84b-wzf49" event={"ID":"9f10f493-9c24-4c41-9a90-4e897969208e","Type":"ContainerDied","Data":"7502fb123098686338fb7b21227895dce01b506e0bed68d08567abe39a8c6657"} Jul 06 20:46:26.301105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.300825 2574 scope.go:117] "RemoveContainer" containerID="06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0" Jul 06 20:46:26.309970 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.309954 2574 scope.go:117] "RemoveContainer" containerID="06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0" Jul 06 20:46:26.310244 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:46:26.310228 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0\": container with ID starting with 06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0 not found: ID does not exist" containerID="06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0" Jul 06 20:46:26.310292 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.310253 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0"} err="failed to get container status \"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0\": rpc error: code = NotFound desc = could not find container \"06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0\": container with ID starting with 06dcf3906eddaeae876caaadd400d9208e4ea8d4357522c32f35415821b329a0 not found: ID does not exist" Jul 06 20:46:26.323585 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.323562 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:46:26.331973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.331955 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6859dcd84b-wzf49"] Jul 06 20:46:26.469994 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.469930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:26.626594 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.626557 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:46:26.630239 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:46:26.630212 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dec807b_15ae_425c_9efd_967d7914b8b0.slice/crio-f9d1fdc750d1c36e890bf0a42b1bfb980fc41d386790809f87501126314897f5 WatchSource:0}: Error finding container f9d1fdc750d1c36e890bf0a42b1bfb980fc41d386790809f87501126314897f5: Status 404 returned error can't find the container with id f9d1fdc750d1c36e890bf0a42b1bfb980fc41d386790809f87501126314897f5 Jul 06 20:46:26.631803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:26.631783 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:46:27.308985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:27.308949 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" event={"ID":"4dec807b-15ae-425c-9efd-967d7914b8b0","Type":"ContainerStarted","Data":"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07"} Jul 06 20:46:27.309412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:27.309005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" event={"ID":"4dec807b-15ae-425c-9efd-967d7914b8b0","Type":"ContainerStarted","Data":"f9d1fdc750d1c36e890bf0a42b1bfb980fc41d386790809f87501126314897f5"} Jul 06 20:46:27.309412 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:27.309034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:27.328939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:27.328869 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" podStartSLOduration=1.870424538 podStartE2EDuration="2.328849188s" totalImagesPullingTime="458.42465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:27.326532366 +0000 UTC m=+1903.817155920" watchObservedRunningTime="2026-07-06 20:46:27.328849188 +0000 UTC m=+1903.819472740" Jul 06 20:46:28.072655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:28.072623 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f10f493-9c24-4c41-9a90-4e897969208e" path="/var/lib/kubelet/pods/9f10f493-9c24-4c41-9a90-4e897969208e/volumes" Jul 06 20:46:29.783779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:29.783747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:30.135351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.135267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:30.426693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.426660 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:46:30.431605 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.431577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.433939 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.433915 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-07298a-openshift-default-dockercfg-dhr5q\"" Jul 06 20:46:30.448020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.447993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:46:30.470511 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.470482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:30.587969 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.587935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.587992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588157 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.588377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.588264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtq7l\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689383 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtq7l\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689452 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689510 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689788 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689765 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.689845 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.689800 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.690105 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.690085 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.691643 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.691625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.691732 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.691687 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.706653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.706628 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.707007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.706989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtq7l\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l\") pod \"e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.741995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.741973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:30.887145 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.887118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:46:30.890013 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:46:30.889980 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef56e5b8_79b1_4759_bb61_967f869b0b06.slice/crio-112f25022b18956965291d1fdea7030e31571cffe84f2ed0c8077a3836c7dcae WatchSource:0}: Error finding container 112f25022b18956965291d1fdea7030e31571cffe84f2ed0c8077a3836c7dcae: Status 404 returned error can't find the container with id 112f25022b18956965291d1fdea7030e31571cffe84f2ed0c8077a3836c7dcae Jul 06 20:46:30.892095 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.892062 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:30.892182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.892127 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:30.892182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:30.892153 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:31.005218 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.005187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:31.332555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.332470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerStarted","Data":"577cc7da330c8704d48e71472e571f8db1b43685b936aad6d7e5bc879fbc19ec"} Jul 06 20:46:31.332555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.332522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerStarted","Data":"112f25022b18956965291d1fdea7030e31571cffe84f2ed0c8077a3836c7dcae"} Jul 06 20:46:31.355042 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.354991 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podStartSLOduration=1.354977679 podStartE2EDuration="1.354977679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:31.353578019 +0000 UTC m=+1907.844201600" watchObservedRunningTime="2026-07-06 20:46:31.354977679 +0000 UTC m=+1907.845601228" Jul 06 20:46:31.742563 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.742529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:31.747519 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:31.747494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:32.338864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:32.338831 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:32.339708 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:32.339690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:33.288956 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.288896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:33.319825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.319798 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:46:33.356137 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.356107 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:46:33.360380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.360355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.362796 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.362773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-07298a-serving-cert\"" Jul 06 20:46:33.370043 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.370020 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:46:33.386087 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.386062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:33.458869 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.458843 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:33.513429 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.513400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96chv\" (UniqueName: \"kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.513589 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.513434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.513589 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.513454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.557473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.557405 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:33.614769 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.614742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96chv\" (UniqueName: \"kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.614895 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.614772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.614895 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.614795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.615512 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.615493 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.617141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.617124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.622969 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.622952 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96chv\" (UniqueName: \"kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv\") pod \"maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:33.677122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:33.677096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:34.021407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:34.021379 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:46:34.022463 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:46:34.022435 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9a04a0_b5f9_4a21_845c_d2b0a8077c38.slice/crio-7eaa736e96f6ca7bc409f4d5c31c4c5461d4d330aa938e7490c5526ff369865f WatchSource:0}: Error finding container 7eaa736e96f6ca7bc409f4d5c31c4c5461d4d330aa938e7490c5526ff369865f: Status 404 returned error can't find the container with id 7eaa736e96f6ca7bc409f4d5c31c4c5461d4d330aa938e7490c5526ff369865f Jul 06 20:46:34.111966 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:34.111935 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:34.212618 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:34.212582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:34.351974 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:34.351870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" event={"ID":"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38","Type":"ContainerStarted","Data":"7eaa736e96f6ca7bc409f4d5c31c4c5461d4d330aa938e7490c5526ff369865f"} Jul 06 20:46:35.358742 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:35.358711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" event={"ID":"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38","Type":"ContainerStarted","Data":"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd"} Jul 06 20:46:35.359135 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:35.358839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:35.378417 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:35.378360 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" podStartSLOduration=2.04648439 podStartE2EDuration="2.378342378s" totalImagesPullingTime="331.857988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:46:35.376103479 +0000 UTC m=+1911.866727030" watchObservedRunningTime="2026-07-06 20:46:35.378342378 +0000 UTC m=+1911.868965934" Jul 06 20:46:37.824315 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:37.818218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:40.328043 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:40.328005 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:46:40.328397 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:40.328119 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:46:41.370220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.370188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:46:41.407319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.405552 2574 generic.go:356] "Generic (PLEG): container finished" podID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerID="93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1" exitCode=0 Jul 06 20:46:41.407319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.405708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerDied","Data":"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1"} Jul 06 20:46:41.407319 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.406225 2574 scope.go:117] "RemoveContainer" containerID="93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1" Jul 06 20:46:41.408740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.408395 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:41.408740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.408485 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:41.408740 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:41.408534 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:42.415614 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:42.415579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerStarted","Data":"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99"} Jul 06 20:46:42.444965 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:42.444930 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:42.445180 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:42.445147 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 06 20:46:42.445263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:42.445235 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 06 20:46:43.445032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:43.444999 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 06 20:46:43.445427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:43.445056 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 06 20:46:44.445189 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:44.445160 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 06 20:46:44.445619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:44.445595 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 06 20:46:45.444308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:45.444279 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 06 20:46:45.444456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:45.444326 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 06 20:46:46.165559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:46.165495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:46.444778 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:46.444699 2574 patch_prober.go:28] interesting pod/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 06 20:46:46.444778 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:46.444750 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 06 20:46:47.448628 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:47.448600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:47.449032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:47.448935 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:47.449665 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:47.449646 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:46:48.066169 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:48.066119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:48.169546 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:48.169512 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:48.380084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:48.379995 2574 patch_prober.go:28] interesting pod/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 20:46:48.380084 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:48.380048 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:46:49.468391 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.468362 2574 generic.go:356] "Generic (PLEG): container finished" podID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerID="577cc7da330c8704d48e71472e571f8db1b43685b936aad6d7e5bc879fbc19ec" exitCode=0 Jul 06 20:46:49.468802 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.468437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerDied","Data":"577cc7da330c8704d48e71472e571f8db1b43685b936aad6d7e5bc879fbc19ec"} Jul 06 20:46:49.469001 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.468985 2574 scope.go:117] "RemoveContainer" containerID="577cc7da330c8704d48e71472e571f8db1b43685b936aad6d7e5bc879fbc19ec" Jul 06 20:46:49.469995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.469964 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:49.470054 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.470035 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:49.470089 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:49.470068 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:46:50.477367 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:50.476416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerStarted","Data":"baef9c6a5112092512f29aa4b245cd2fa7241fac18d7bb983be6210cfaaf9b49"} Jul 06 20:46:50.742785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:50.742696 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:50.742983 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:50.742945 2574 patch_prober.go:28] interesting pod/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 06 20:46:50.743058 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:50.743002 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 06 20:46:51.743174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:51.743143 2574 patch_prober.go:28] interesting pod/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 06 20:46:51.743559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:51.743205 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 06 20:46:52.742370 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:52.742337 2574 patch_prober.go:28] interesting pod/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 06 20:46:52.742619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:52.742389 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 06 20:46:53.742706 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:53.742669 2574 patch_prober.go:28] interesting pod/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 06 20:46:53.743112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:53.742725 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 06 20:46:54.745958 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:54.745928 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:54.746368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:54.746281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:54.747011 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:54.746989 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:46:55.288254 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:55.288219 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:46:55.386543 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:46:55.386511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:00.271534 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:00.271505 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:00.342390 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:00.342356 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:17.007714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:17.007682 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:36.512474 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:36.512384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:37.398751 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:37.398715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:37.586143 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:37.586113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:37.996588 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:37.996547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:38.108789 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.108762 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:38.215464 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.215436 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:38.356695 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.356618 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:47:38.356859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.356835 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" podUID="4dec807b-15ae-425c-9efd-967d7914b8b0" containerName="maas-api" containerID="cri-o://153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07" gracePeriod=30 Jul 06 20:47:38.392503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.392471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:38.612075 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.612024 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:47:38.725794 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.725765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") pod \"4dec807b-15ae-425c-9efd-967d7914b8b0\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " Jul 06 20:47:38.725974 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.725836 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca\") pod \"4dec807b-15ae-425c-9efd-967d7914b8b0\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " Jul 06 20:47:38.725974 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.725878 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88\") pod \"4dec807b-15ae-425c-9efd-967d7914b8b0\" (UID: \"4dec807b-15ae-425c-9efd-967d7914b8b0\") " Jul 06 20:47:38.726222 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.726200 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4dec807b-15ae-425c-9efd-967d7914b8b0" (UID: "4dec807b-15ae-425c-9efd-967d7914b8b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:47:38.727732 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.727705 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4dec807b-15ae-425c-9efd-967d7914b8b0" (UID: "4dec807b-15ae-425c-9efd-967d7914b8b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:47:38.727980 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.727957 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88" (OuterVolumeSpecName: "kube-api-access-zqd88") pod "4dec807b-15ae-425c-9efd-967d7914b8b0" (UID: "4dec807b-15ae-425c-9efd-967d7914b8b0"). InnerVolumeSpecName "kube-api-access-zqd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:47:38.783815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.783781 2574 generic.go:356] "Generic (PLEG): container finished" podID="4dec807b-15ae-425c-9efd-967d7914b8b0" containerID="153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07" exitCode=0 Jul 06 20:47:38.783973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.783855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" event={"ID":"4dec807b-15ae-425c-9efd-967d7914b8b0","Type":"ContainerDied","Data":"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07"} Jul 06 20:47:38.783973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.783863 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" Jul 06 20:47:38.783973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.783953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q" event={"ID":"4dec807b-15ae-425c-9efd-967d7914b8b0","Type":"ContainerDied","Data":"f9d1fdc750d1c36e890bf0a42b1bfb980fc41d386790809f87501126314897f5"} Jul 06 20:47:38.784148 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.783981 2574 scope.go:117] "RemoveContainer" containerID="153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07" Jul 06 20:47:38.794344 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.794326 2574 scope.go:117] "RemoveContainer" containerID="153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07" Jul 06 20:47:38.794646 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:47:38.794628 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07\": container with ID starting with 153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07 not found: ID does not exist" containerID="153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07" Jul 06 20:47:38.794712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.794658 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07"} err="failed to get container status \"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07\": rpc error: code = NotFound desc = could not find container \"153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07\": container with ID starting with 153d49e6ba02d3dae55dcf76dc31c56b3199c7f07217eda6935c395e64db8b07 not found: ID does not exist" Jul 06 20:47:38.810495 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.810464 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:47:38.816257 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.816238 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-37baf2-f97d769f8-8f48q"] Jul 06 20:47:38.826813 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.826796 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dec807b-15ae-425c-9efd-967d7914b8b0-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.826887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.826816 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dec807b-15ae-425c-9efd-967d7914b8b0-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:47:38.826887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:38.826826 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/4dec807b-15ae-425c-9efd-967d7914b8b0-kube-api-access-zqd88\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:47:40.073215 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:40.073180 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dec807b-15ae-425c-9efd-967d7914b8b0" path="/var/lib/kubelet/pods/4dec807b-15ae-425c-9efd-967d7914b8b0/volumes" Jul 06 20:47:43.053018 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:43.052987 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:59.565102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:59.565064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:47:59.666608 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:47:59.666571 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:20.422759 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:20.422698 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:20.716745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:20.716669 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:48:20.716883 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:20.716859 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" containerID="cri-o://d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99" gracePeriod=30 Jul 06 20:48:20.757415 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:20.757384 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:20.975554 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:20.975475 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:21.175418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.175381 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:21.549422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.549394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:21.666056 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.666015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:21.785122 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.785091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:21.953918 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.953884 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:48:21.954129 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:21.954108 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" podUID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" containerName="maas-api" containerID="cri-o://b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd" gracePeriod=30 Jul 06 20:48:22.000667 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.000641 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:22.210280 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.210224 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:48:22.393220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.393198 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca\") pod \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " Jul 06 20:48:22.393359 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.393236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls\") pod \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " Jul 06 20:48:22.393359 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.393282 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96chv\" (UniqueName: \"kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv\") pod \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\" (UID: \"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38\") " Jul 06 20:48:22.393534 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.393509 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" (UID: "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:22.395245 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.395212 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" (UID: "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:48:22.395364 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.395341 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv" (OuterVolumeSpecName: "kube-api-access-96chv") pod "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" (UID: "4a9a04a0-b5f9-4a21-845c-d2b0a8077c38"). InnerVolumeSpecName "kube-api-access-96chv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:22.494427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.494372 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96chv\" (UniqueName: \"kubernetes.io/projected/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-kube-api-access-96chv\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:22.494427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.494401 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:22.494427 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:22.494416 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:23.033234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.033200 2574 generic.go:356] "Generic (PLEG): container finished" podID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" containerID="b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd" exitCode=0 Jul 06 20:48:23.033655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.033243 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" Jul 06 20:48:23.033655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.033279 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" event={"ID":"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38","Type":"ContainerDied","Data":"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd"} Jul 06 20:48:23.033655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.033329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx" event={"ID":"4a9a04a0-b5f9-4a21-845c-d2b0a8077c38","Type":"ContainerDied","Data":"7eaa736e96f6ca7bc409f4d5c31c4c5461d4d330aa938e7490c5526ff369865f"} Jul 06 20:48:23.033655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.033351 2574 scope.go:117] "RemoveContainer" containerID="b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd" Jul 06 20:48:23.042881 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.042864 2574 scope.go:117] "RemoveContainer" containerID="b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd" Jul 06 20:48:23.043105 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:23.043088 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd\": container with ID starting with b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd not found: ID does not exist" containerID="b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd" Jul 06 20:48:23.043150 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.043110 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd"} err="failed to get container status \"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd\": rpc error: code = NotFound desc = could not find container \"b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd\": container with ID starting with b4e59201d7ee43f2544f6110897ef95c9adf7d6d87d19d6e123f83c3529603fd not found: ID does not exist" Jul 06 20:48:23.056304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.056281 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:48:23.061783 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:23.061763 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-07298a-6485dc6867-4lqkx"] Jul 06 20:48:24.071713 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:24.071683 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" path="/var/lib/kubelet/pods/4a9a04a0-b5f9-4a21-845c-d2b0a8077c38/volumes" Jul 06 20:48:25.974298 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:25.974272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:48:26.020455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020441 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020492 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020768 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020806 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27mb9\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9\") pod \"587583a4-b03d-4418-a3ec-326b872ce9ab\" (UID: \"587583a4-b03d-4418-a3ec-326b872ce9ab\") " Jul 06 20:48:26.020861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.020838 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:26.021086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021018 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:26.021086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021042 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:26.021252 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021232 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.021418 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021403 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.021559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021537 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.021559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.021268 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:26.022120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.022097 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:26.022849 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.022820 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:26.023134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.023113 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:48:26.023233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.023151 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:26.023289 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.023225 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9" (OuterVolumeSpecName: "kube-api-access-27mb9") pod "587583a4-b03d-4418-a3ec-326b872ce9ab" (UID: "587583a4-b03d-4418-a3ec-326b872ce9ab"). InnerVolumeSpecName "kube-api-access-27mb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:26.052802 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.052777 2574 generic.go:356] "Generic (PLEG): container finished" podID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerID="d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99" exitCode=0 Jul 06 20:48:26.052891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.052819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerDied","Data":"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99"} Jul 06 20:48:26.052891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.052827 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" Jul 06 20:48:26.052891 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.052873 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6" event={"ID":"587583a4-b03d-4418-a3ec-326b872ce9ab","Type":"ContainerDied","Data":"8c03b3be76c879ed0f3ac7d2bbcb0782faf96290cfba9d6cb83aec814cdf5430"} Jul 06 20:48:26.053033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.052929 2574 scope.go:117] "RemoveContainer" containerID="d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99" Jul 06 20:48:26.062453 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.062438 2574 scope.go:117] "RemoveContainer" containerID="93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1" Jul 06 20:48:26.071246 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.071231 2574 scope.go:117] "RemoveContainer" containerID="d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99" Jul 06 20:48:26.071514 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:26.071495 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99\": container with ID starting with d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99 not found: ID does not exist" containerID="d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99" Jul 06 20:48:26.071555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.071523 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99"} err="failed to get container status \"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99\": rpc error: code = NotFound desc = could not find container \"d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99\": container with ID starting with d6dc9c7d7d5fda1eac384ad544967dc47a8755dc8d2791d35929f017c6c29c99 not found: ID does not exist" Jul 06 20:48:26.071555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.071540 2574 scope.go:117] "RemoveContainer" containerID="93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1" Jul 06 20:48:26.071798 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:26.071780 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1\": container with ID starting with 93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1 not found: ID does not exist" containerID="93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1" Jul 06 20:48:26.071853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.071808 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1"} err="failed to get container status \"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1\": rpc error: code = NotFound desc = could not find container \"93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1\": container with ID starting with 93666e8e6deac07a411a1608529c100d902ecf134f053ee6c93a6aa55cbe4de1 not found: ID does not exist" Jul 06 20:48:26.077435 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.077412 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:48:26.083608 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.083585 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-37baf2-openshift-default-77bf464d55-jqmd6"] Jul 06 20:48:26.122124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122101 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mb9\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-kube-api-access-27mb9\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.122124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122122 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.122287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122132 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587583a4-b03d-4418-a3ec-326b872ce9ab-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.122287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122160 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.122287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122182 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:26.122287 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:26.122194 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587583a4-b03d-4418-a3ec-326b872ce9ab-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:27.654581 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:27.654547 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:27.824835 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:27.824805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:27.975467 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:27.975434 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:48:27.975701 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:27.975673 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" containerID="cri-o://baef9c6a5112092512f29aa4b245cd2fa7241fac18d7bb983be6210cfaaf9b49" gracePeriod=30 Jul 06 20:48:28.013518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.013489 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:28.072294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.072249 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" path="/var/lib/kubelet/pods/587583a4-b03d-4418-a3ec-326b872ce9ab/volumes" Jul 06 20:48:28.211848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.211805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:28.565144 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.565098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:28.939699 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.939668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:48:28.940134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940119 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940135 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940148 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" containerName="maas-api" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940155 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" containerName="maas-api" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940161 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940167 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940178 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940179 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dec807b-15ae-425c-9efd-967d7914b8b0" containerName="maas-api" Jul 06 20:48:28.940365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940185 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dec807b-15ae-425c-9efd-967d7914b8b0" containerName="maas-api" Jul 06 20:48:28.940365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940246 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940253 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9a04a0-b5f9-4a21-845c-d2b0a8077c38" containerName="maas-api" Jul 06 20:48:28.940365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940260 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="587583a4-b03d-4418-a3ec-326b872ce9ab" containerName="istio-proxy" Jul 06 20:48:28.940365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.940267 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dec807b-15ae-425c-9efd-967d7914b8b0" containerName="maas-api" Jul 06 20:48:28.943511 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.943495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:28.946866 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.946849 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-220e12-openshift-default-dockercfg-699dn\"" Jul 06 20:48:28.977774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:28.977753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:48:29.046072 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhjt\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046283 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.046485 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.046452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.074135 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.074093 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:29.147714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.147889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhjt\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.147889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.147889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.147889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.147889 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.147987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148139 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148214 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148191 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148355 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.148768 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.148745 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.150316 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.150292 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.150519 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.150501 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.160774 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.160754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.162029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.162006 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhjt\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt\") pod \"e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.175467 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.175447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:29.253657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.253602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:29.399182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.399151 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:48:29.400312 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:48:29.400289 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95db0404_88e4_452d_9bd6_668dbfab71c7.slice/crio-6f8be217e1d892cd5e81df62473a8e193f2be029b4247e6119749bdfaace70da WatchSource:0}: Error finding container 6f8be217e1d892cd5e81df62473a8e193f2be029b4247e6119749bdfaace70da: Status 404 returned error can't find the container with id 6f8be217e1d892cd5e81df62473a8e193f2be029b4247e6119749bdfaace70da Jul 06 20:48:29.403019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.402984 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:29.403124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.403064 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:29.403124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:29.403107 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:30.083615 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:30.083579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerStarted","Data":"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb"} Jul 06 20:48:30.084032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:30.083641 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerStarted","Data":"6f8be217e1d892cd5e81df62473a8e193f2be029b4247e6119749bdfaace70da"} Jul 06 20:48:30.114700 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:30.114643 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podStartSLOduration=2.114626616 podStartE2EDuration="2.114626616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:30.113892966 +0000 UTC m=+2026.604516527" watchObservedRunningTime="2026-07-06 20:48:30.114626616 +0000 UTC m=+2026.605250169" Jul 06 20:48:30.254204 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:30.254164 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:30.258803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:30.258779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:31.089297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:31.089261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:31.090075 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:31.090058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:32.107458 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.107425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:32.201395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.201352 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:32.284365 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.284329 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:32.288055 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.288034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.295733 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.295713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-220e12-serving-cert\"" Jul 06 20:48:32.309769 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.309747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:32.383174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.383088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.383174 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.383144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crlj\" (UniqueName: \"kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.383374 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.383233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.386825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.386774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:32.484189 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.484155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.484372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.484193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crlj\" (UniqueName: \"kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.484372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.484239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.484372 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:32.484298 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-220e12-serving-cert: secret "maas-api-e2e-api-a-220e12-serving-cert" not found Jul 06 20:48:32.484525 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:32.484391 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls podName:aca0c0f1-f9d1-4a55-908e-47e01f77ee6c nodeName:}" failed. No retries permitted until 2026-07-06 20:48:32.984375041 +0000 UTC m=+2029.474998568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls") pod "maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" (UID: "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c") : secret "maas-api-e2e-api-a-220e12-serving-cert" not found Jul 06 20:48:32.484839 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.484820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.495834 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.495808 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crlj\" (UniqueName: \"kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.608940 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.608916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:32.988733 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.988700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:32.991349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:32.991328 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") pod \"maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:33.007253 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.007228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:33.106304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.106270 2574 generic.go:356] "Generic (PLEG): container finished" podID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerID="baef9c6a5112092512f29aa4b245cd2fa7241fac18d7bb983be6210cfaaf9b49" exitCode=0 Jul 06 20:48:33.106444 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.106340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerDied","Data":"baef9c6a5112092512f29aa4b245cd2fa7241fac18d7bb983be6210cfaaf9b49"} Jul 06 20:48:33.106444 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.106402 2574 scope.go:117] "RemoveContainer" containerID="577cc7da330c8704d48e71472e571f8db1b43685b936aad6d7e5bc879fbc19ec" Jul 06 20:48:33.121393 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.121367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:33.199856 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.199827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:33.237052 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.236995 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:48:33.291153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291298 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291298 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291214 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291298 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291310 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtq7l\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291479 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291395 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291478 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291508 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token\") pod \"ef56e5b8-79b1-4759-bb61-967f869b0b06\" (UID: \"ef56e5b8-79b1-4759-bb61-967f869b0b06\") " Jul 06 20:48:33.291633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291332 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:33.291633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291519 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:33.291633 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291551 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:33.291950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291783 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:33.291950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291835 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.291950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291854 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.291950 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.291869 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.292256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.292234 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:33.294368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.294335 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:48:33.294368 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.294355 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l" (OuterVolumeSpecName: "kube-api-access-wtq7l") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "kube-api-access-wtq7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:33.294555 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.294531 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:48:33.295556 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.295531 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef56e5b8-79b1-4759-bb61-967f869b0b06" (UID: "ef56e5b8-79b1-4759-bb61-967f869b0b06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:33.364120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.364094 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:33.364794 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:48:33.364772 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca0c0f1_f9d1_4a55_908e_47e01f77ee6c.slice/crio-dc3c2249f8e54e63f8279e6d543ed723977701be5416a717671ae4064bd61fa7 WatchSource:0}: Error finding container dc3c2249f8e54e63f8279e6d543ed723977701be5416a717671ae4064bd61fa7: Status 404 returned error can't find the container with id dc3c2249f8e54e63f8279e6d543ed723977701be5416a717671ae4064bd61fa7 Jul 06 20:48:33.392715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392693 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.392715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392714 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef56e5b8-79b1-4759-bb61-967f869b0b06-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.392848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392725 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtq7l\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-kube-api-access-wtq7l\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.392848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392734 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.392848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392741 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:33.392848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:33.392748 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef56e5b8-79b1-4759-bb61-967f869b0b06-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:34.113725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.113685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" event={"ID":"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c","Type":"ContainerStarted","Data":"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988"} Jul 06 20:48:34.113894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.113824 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:34.113894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.113881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" event={"ID":"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c","Type":"ContainerStarted","Data":"dc3c2249f8e54e63f8279e6d543ed723977701be5416a717671ae4064bd61fa7"} Jul 06 20:48:34.115136 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.115118 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" Jul 06 20:48:34.115256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.115144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9" event={"ID":"ef56e5b8-79b1-4759-bb61-967f869b0b06","Type":"ContainerDied","Data":"112f25022b18956965291d1fdea7030e31571cffe84f2ed0c8077a3836c7dcae"} Jul 06 20:48:34.115256 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.115184 2574 scope.go:117] "RemoveContainer" containerID="baef9c6a5112092512f29aa4b245cd2fa7241fac18d7bb983be6210cfaaf9b49" Jul 06 20:48:34.152657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.152614 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" podStartSLOduration=1.728772196 podStartE2EDuration="2.152601991s" totalImagesPullingTime="423.829795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:34.150473223 +0000 UTC m=+2030.641096777" watchObservedRunningTime="2026-07-06 20:48:34.152601991 +0000 UTC m=+2030.643225541" Jul 06 20:48:34.169878 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.169851 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:48:34.174684 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:34.174657 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-07298a-openshift-default-5d59f6f6df-npbf9"] Jul 06 20:48:36.072780 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:36.072743 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" path="/var/lib/kubelet/pods/ef56e5b8-79b1-4759-bb61-967f869b0b06/volumes" Jul 06 20:48:36.564120 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:36.564083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:37.062562 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.062529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:37.397257 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.397178 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:48:37.397619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.397589 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:37.397619 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.397601 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:37.397697 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.397658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:37.402192 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.402172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.405263 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.405245 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b42d16-openshift-default-dockercfg-58tdm\"" Jul 06 20:48:37.421635 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.421613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:48:37.439093 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.439068 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:37.529819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.529788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530035 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.529836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530035 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.529869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530035 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.529893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530035 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.529998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6pv\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.530042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.530088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.530111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.530235 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.530160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.564016 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.563990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:37.631069 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6pv\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631425 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631680 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631734 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631664 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631781 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631874 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631814 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.631945 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.631920 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.633429 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.633409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.633679 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.633659 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.641062 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.641040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.641191 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.641107 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6pv\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv\") pod \"e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.712683 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.712658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:37.862151 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.862125 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:48:37.864031 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:48:37.864004 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab14378b_bce7_4f69_a072_97e5e6d2fad0.slice/crio-9fa58f287269a4d451e95466a070ef65593c24994f76654a8fcb6a82f46bd56d WatchSource:0}: Error finding container 9fa58f287269a4d451e95466a070ef65593c24994f76654a8fcb6a82f46bd56d: Status 404 returned error can't find the container with id 9fa58f287269a4d451e95466a070ef65593c24994f76654a8fcb6a82f46bd56d Jul 06 20:48:37.866232 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.866202 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:37.866301 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.866271 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:37.866301 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:37.866298 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:38.140305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:38.140208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerStarted","Data":"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484"} Jul 06 20:48:38.140305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:38.140276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerStarted","Data":"9fa58f287269a4d451e95466a070ef65593c24994f76654a8fcb6a82f46bd56d"} Jul 06 20:48:38.211566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:38.211505 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podStartSLOduration=1.211481992 podStartE2EDuration="1.211481992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:38.211045211 +0000 UTC m=+2034.701668791" watchObservedRunningTime="2026-07-06 20:48:38.211481992 +0000 UTC m=+2034.702105546" Jul 06 20:48:38.713841 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:38.713806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:38.718797 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:38.718771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:39.145675 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:39.145589 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:39.146474 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:39.146450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:40.128457 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.128425 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:40.316271 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.316231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:40.406381 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.406344 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:48:40.407060 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.407040 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:40.407231 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.407215 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:40.407486 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.407468 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef56e5b8-79b1-4759-bb61-967f869b0b06" containerName="istio-proxy" Jul 06 20:48:40.411559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.411541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.414347 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.414325 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-b42d16-serving-cert\"" Jul 06 20:48:40.428217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.428195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:48:40.446194 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.446171 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:40.460379 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.460354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.460488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.460396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdbc\" (UniqueName: \"kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.460488 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.460435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.561789 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.561754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.561996 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.561808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdbc\" (UniqueName: \"kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.561996 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.561850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.561996 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:40.561954 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-b42d16-serving-cert: secret "maas-api-e2e-api-b-b42d16-serving-cert" not found Jul 06 20:48:40.562111 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:40.562033 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls podName:0ceba5da-f034-4f92-af25-5f6f6df453c6 nodeName:}" failed. No retries permitted until 2026-07-06 20:48:41.062013716 +0000 UTC m=+2037.552637244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls") pod "maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" (UID: "0ceba5da-f034-4f92-af25-5f6f6df453c6") : secret "maas-api-e2e-api-b-b42d16-serving-cert" not found Jul 06 20:48:40.562549 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.562532 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.569864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.569832 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:40.586222 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.586200 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdbc\" (UniqueName: \"kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:40.709072 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:40.708976 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:41.066829 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.066739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:41.069342 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.069320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") pod \"maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:41.085114 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.085081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:41.185489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.185440 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:41.322861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.322782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:41.514102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:41.512650 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:48:41.516300 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:48:41.516268 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ceba5da_f034_4f92_af25_5f6f6df453c6.slice/crio-cb4b42e8007bbdef14624cb6e0728fedebc15556fb165d5d85739312ed4a6192 WatchSource:0}: Error finding container cb4b42e8007bbdef14624cb6e0728fedebc15556fb165d5d85739312ed4a6192: Status 404 returned error can't find the container with id cb4b42e8007bbdef14624cb6e0728fedebc15556fb165d5d85739312ed4a6192 Jul 06 20:48:42.172431 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:42.172393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" event={"ID":"0ceba5da-f034-4f92-af25-5f6f6df453c6","Type":"ContainerStarted","Data":"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469"} Jul 06 20:48:42.172540 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:42.172491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" event={"ID":"0ceba5da-f034-4f92-af25-5f6f6df453c6","Type":"ContainerStarted","Data":"cb4b42e8007bbdef14624cb6e0728fedebc15556fb165d5d85739312ed4a6192"} Jul 06 20:48:42.172540 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:42.172526 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:42.201549 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:42.201488 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" podStartSLOduration=1.632885923 podStartE2EDuration="2.20146687s" totalImagesPullingTime="568.580947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:48:42.200143006 +0000 UTC m=+2038.690766559" watchObservedRunningTime="2026-07-06 20:48:42.20146687 +0000 UTC m=+2038.692090421" Jul 06 20:48:44.694434 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:44.694385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:45.198745 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.198702 2574 generic.go:356] "Generic (PLEG): container finished" podID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerID="df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb" exitCode=0 Jul 06 20:48:45.199020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.198872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerDied","Data":"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb"} Jul 06 20:48:45.199406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.199383 2574 scope.go:117] "RemoveContainer" containerID="df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb" Jul 06 20:48:45.201844 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.201798 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:45.201972 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.201897 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:45.201972 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:45.201967 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:46.089574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:46.089483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:46.215871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:46.215802 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerStarted","Data":"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f"} Jul 06 20:48:46.254258 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:46.254231 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:46.255163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:46.255143 2574 patch_prober.go:28] interesting pod/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:48:46.255324 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:46.255306 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 06 20:48:47.254070 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:47.254038 2574 patch_prober.go:28] interesting pod/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:48:47.254441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:47.254096 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 06 20:48:48.188700 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:48.187836 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:48:48.254564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:48.254535 2574 patch_prober.go:28] interesting pod/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:48:48.254937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:48.254615 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 06 20:48:49.254146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:49.254064 2574 patch_prober.go:28] interesting pod/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Jul 06 20:48:49.254146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:49.254115 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Jul 06 20:48:50.258792 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:50.258761 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:50.259269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:50.259159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:50.260005 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:50.259986 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:48:52.266366 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.266300 2574 generic.go:356] "Generic (PLEG): container finished" podID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerID="8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484" exitCode=0 Jul 06 20:48:52.266814 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.266369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerDied","Data":"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484"} Jul 06 20:48:52.266934 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.266896 2574 scope.go:117] "RemoveContainer" containerID="8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484" Jul 06 20:48:52.267896 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.267861 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:52.267989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.267953 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:52.268037 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.267988 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:48:52.872896 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:52.872862 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:53.261369 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.261335 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:53.273968 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.273936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerStarted","Data":"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f"} Jul 06 20:48:53.384973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.384833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:53.484394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.484308 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:53.662447 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.662414 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:53.662998 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.662958 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" podUID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" containerName="maas-api" containerID="cri-o://37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988" gracePeriod=30 Jul 06 20:48:53.705980 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.700304 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:53.713567 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.713492 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:53.715021 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.714724 2574 patch_prober.go:28] interesting pod/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 06 20:48:53.715021 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.714771 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 06 20:48:53.960987 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:53.960960 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:54.114405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.114370 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca\") pod \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " Jul 06 20:48:54.114405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.114411 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6crlj\" (UniqueName: \"kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj\") pod \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " Jul 06 20:48:54.114618 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.114450 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") pod \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\" (UID: \"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c\") " Jul 06 20:48:54.114768 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.114744 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" (UID: "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:48:54.116514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.116490 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj" (OuterVolumeSpecName: "kube-api-access-6crlj") pod "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" (UID: "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c"). InnerVolumeSpecName "kube-api-access-6crlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:48:54.116514 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.116504 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" (UID: "aca0c0f1-f9d1-4a55-908e-47e01f77ee6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:48:54.145780 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.145756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:54.216014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.215931 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:54.216014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.215958 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crlj\" (UniqueName: \"kubernetes.io/projected/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-kube-api-access-6crlj\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:54.216014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.215971 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:48:54.283602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.283564 2574 generic.go:356] "Generic (PLEG): container finished" podID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" containerID="37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988" exitCode=0 Jul 06 20:48:54.284132 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.284107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" event={"ID":"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c","Type":"ContainerDied","Data":"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988"} Jul 06 20:48:54.284237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.284146 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" Jul 06 20:48:54.284237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.284162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8" event={"ID":"aca0c0f1-f9d1-4a55-908e-47e01f77ee6c","Type":"ContainerDied","Data":"dc3c2249f8e54e63f8279e6d543ed723977701be5416a717671ae4064bd61fa7"} Jul 06 20:48:54.284237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.284191 2574 scope.go:117] "RemoveContainer" containerID="37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988" Jul 06 20:48:54.316476 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.316450 2574 scope.go:117] "RemoveContainer" containerID="37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988" Jul 06 20:48:54.316864 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:48:54.316837 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988\": container with ID starting with 37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988 not found: ID does not exist" containerID="37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988" Jul 06 20:48:54.316993 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.316869 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988"} err="failed to get container status \"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988\": rpc error: code = NotFound desc = could not find container \"37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988\": container with ID starting with 37f2bc2c5102e2ab44a4ac01468f1e29f3ba4084da3a615176fec24796846988 not found: ID does not exist" Jul 06 20:48:54.331696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.331675 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:54.336003 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.335977 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-220e12-676d65fb6d-rqqd8"] Jul 06 20:48:54.713293 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.713260 2574 patch_prober.go:28] interesting pod/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 06 20:48:54.713458 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:54.713314 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 06 20:48:55.714033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:55.713994 2574 patch_prober.go:28] interesting pod/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 06 20:48:55.714423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:55.714054 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 06 20:48:56.072961 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:56.072861 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" path="/var/lib/kubelet/pods/aca0c0f1-f9d1-4a55-908e-47e01f77ee6c/volumes" Jul 06 20:48:56.713265 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:56.713228 2574 patch_prober.go:28] interesting pod/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Jul 06 20:48:56.713503 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:56.713288 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Jul 06 20:48:57.716559 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:57.716529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:57.717523 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:57.717502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:48:59.230725 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.230688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:59.312626 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.312585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:59.507441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.507366 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:48:59.507592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.507559 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" containerID="cri-o://cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f" gracePeriod=30 Jul 06 20:48:59.531865 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.531841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:59.746352 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.746317 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:48:59.935155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:48:59.935116 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:00.369777 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.369688 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:00.468405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.468372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:00.569976 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.569946 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:00.714598 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.714564 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:49:00.714860 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.714829 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" podUID="0ceba5da-f034-4f92-af25-5f6f6df453c6" containerName="maas-api" containerID="cri-o://cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469" gracePeriod=30 Jul 06 20:49:00.732441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.732406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:00.960582 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.960558 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:49:00.973757 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.973710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") pod \"0ceba5da-f034-4f92-af25-5f6f6df453c6\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " Jul 06 20:49:00.973855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.973758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca\") pod \"0ceba5da-f034-4f92-af25-5f6f6df453c6\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " Jul 06 20:49:00.973855 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.973798 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvdbc\" (UniqueName: \"kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc\") pod \"0ceba5da-f034-4f92-af25-5f6f6df453c6\" (UID: \"0ceba5da-f034-4f92-af25-5f6f6df453c6\") " Jul 06 20:49:00.974118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.974092 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ceba5da-f034-4f92-af25-5f6f6df453c6" (UID: "0ceba5da-f034-4f92-af25-5f6f6df453c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:49:00.975620 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.975591 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ceba5da-f034-4f92-af25-5f6f6df453c6" (UID: "0ceba5da-f034-4f92-af25-5f6f6df453c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:49:00.975713 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:00.975689 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc" (OuterVolumeSpecName: "kube-api-access-gvdbc") pod "0ceba5da-f034-4f92-af25-5f6f6df453c6" (UID: "0ceba5da-f034-4f92-af25-5f6f6df453c6"). InnerVolumeSpecName "kube-api-access-gvdbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:49:01.074806 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.074778 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ceba5da-f034-4f92-af25-5f6f6df453c6-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:01.074806 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.074804 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ceba5da-f034-4f92-af25-5f6f6df453c6-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:01.074979 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.074813 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdbc\" (UniqueName: \"kubernetes.io/projected/0ceba5da-f034-4f92-af25-5f6f6df453c6-kube-api-access-gvdbc\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:01.330550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.330476 2574 generic.go:356] "Generic (PLEG): container finished" podID="0ceba5da-f034-4f92-af25-5f6f6df453c6" containerID="cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469" exitCode=0 Jul 06 20:49:01.330550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.330526 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" Jul 06 20:49:01.330550 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.330537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" event={"ID":"0ceba5da-f034-4f92-af25-5f6f6df453c6","Type":"ContainerDied","Data":"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469"} Jul 06 20:49:01.330770 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.330576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2" event={"ID":"0ceba5da-f034-4f92-af25-5f6f6df453c6","Type":"ContainerDied","Data":"cb4b42e8007bbdef14624cb6e0728fedebc15556fb165d5d85739312ed4a6192"} Jul 06 20:49:01.330770 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.330596 2574 scope.go:117] "RemoveContainer" containerID="cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469" Jul 06 20:49:01.340496 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.340473 2574 scope.go:117] "RemoveContainer" containerID="cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469" Jul 06 20:49:01.340800 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:01.340778 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469\": container with ID starting with cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469 not found: ID does not exist" containerID="cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469" Jul 06 20:49:01.340859 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.340811 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469"} err="failed to get container status \"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469\": rpc error: code = NotFound desc = could not find container \"cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469\": container with ID starting with cc812304eacfb0fb751bafe6bcbe8f2c702449b80a3b63e2d73853f4ba298469 not found: ID does not exist" Jul 06 20:49:01.353703 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.353679 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:49:01.359986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:01.359967 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-b42d16-5d9c4f9596-mndg2"] Jul 06 20:49:02.072915 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:02.072868 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ceba5da-f034-4f92-af25-5f6f6df453c6" path="/var/lib/kubelet/pods/0ceba5da-f034-4f92-af25-5f6f6df453c6/volumes" Jul 06 20:49:04.765475 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.765453 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:49:04.806126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806126 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806145 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806199 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806219 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rhjt\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806271 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806300 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806299 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo\") pod \"95db0404-88e4-452d-9bd6-668dbfab71c7\" (UID: \"95db0404-88e4-452d-9bd6-668dbfab71c7\") " Jul 06 20:49:04.806652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806435 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:04.806652 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806570 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:04.806817 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806690 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:49:04.806817 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806781 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.806817 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.806801 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.807206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.807165 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:04.807308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.807212 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:04.808516 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.808473 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:04.808819 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.808794 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:49:04.809002 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.808976 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt" (OuterVolumeSpecName: "kube-api-access-5rhjt") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "kube-api-access-5rhjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:49:04.809320 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.809296 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "95db0404-88e4-452d-9bd6-668dbfab71c7" (UID: "95db0404-88e4-452d-9bd6-668dbfab71c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:49:04.907356 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907333 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907356 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907354 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907362 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907370 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95db0404-88e4-452d-9bd6-668dbfab71c7-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907380 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rhjt\" (UniqueName: \"kubernetes.io/projected/95db0404-88e4-452d-9bd6-668dbfab71c7-kube-api-access-5rhjt\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907388 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95db0404-88e4-452d-9bd6-668dbfab71c7-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:04.907489 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:04.907397 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95db0404-88e4-452d-9bd6-668dbfab71c7-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:05.356258 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.356225 2574 generic.go:356] "Generic (PLEG): container finished" podID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerID="cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f" exitCode=0 Jul 06 20:49:05.356455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.356275 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" Jul 06 20:49:05.356455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.356284 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerDied","Data":"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f"} Jul 06 20:49:05.356455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.356324 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5" event={"ID":"95db0404-88e4-452d-9bd6-668dbfab71c7","Type":"ContainerDied","Data":"6f8be217e1d892cd5e81df62473a8e193f2be029b4247e6119749bdfaace70da"} Jul 06 20:49:05.356455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.356345 2574 scope.go:117] "RemoveContainer" containerID="cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f" Jul 06 20:49:05.366149 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.366132 2574 scope.go:117] "RemoveContainer" containerID="df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb" Jul 06 20:49:05.374279 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.374262 2574 scope.go:117] "RemoveContainer" containerID="cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f" Jul 06 20:49:05.374529 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:05.374510 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f\": container with ID starting with cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f not found: ID does not exist" containerID="cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f" Jul 06 20:49:05.374581 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.374537 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f"} err="failed to get container status \"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f\": rpc error: code = NotFound desc = could not find container \"cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f\": container with ID starting with cc824b86f7c90b38ce3d71a656e9dfb12e7abf109e27bbc6dbd4bd29d7578d4f not found: ID does not exist" Jul 06 20:49:05.374581 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.374556 2574 scope.go:117] "RemoveContainer" containerID="df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb" Jul 06 20:49:05.374794 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:05.374777 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb\": container with ID starting with df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb not found: ID does not exist" containerID="df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb" Jul 06 20:49:05.374832 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.374800 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb"} err="failed to get container status \"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb\": rpc error: code = NotFound desc = could not find container \"df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb\": container with ID starting with df010fe2189f51d3ecfa210a157f816c4f43da0e023665aba242e307e689e6fb not found: ID does not exist" Jul 06 20:49:05.382297 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.382273 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:49:05.389233 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:05.389215 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-220e12-openshift-default-6986f77c4c-42tm5"] Jul 06 20:49:06.072825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.072738 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" path="/var/lib/kubelet/pods/95db0404-88e4-452d-9bd6-668dbfab71c7/volumes" Jul 06 20:49:06.453020 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.452986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:06.549872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.549837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:06.699518 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.699490 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:49:06.699721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.699686 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" containerID="cri-o://d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f" gracePeriod=30 Jul 06 20:49:06.739551 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:06.739468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:07.021576 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.021473 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:07.337285 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.337188 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:07.646110 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646035 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:49:07.646421 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646409 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ceba5da-f034-4f92-af25-5f6f6df453c6" containerName="maas-api" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646422 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ceba5da-f034-4f92-af25-5f6f6df453c6" containerName="maas-api" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646433 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646439 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646445 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" containerName="maas-api" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646451 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" containerName="maas-api" Jul 06 20:49:07.646469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646468 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:07.646658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646474 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:07.646658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646531 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:07.646658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646540 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca0c0f1-f9d1-4a55-908e-47e01f77ee6c" containerName="maas-api" Jul 06 20:49:07.646658 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.646550 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ceba5da-f034-4f92-af25-5f6f6df453c6" containerName="maas-api" Jul 06 20:49:07.649521 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.649500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.652115 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.652096 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7306a9-openshift-default-dockercfg-9x9wj\"" Jul 06 20:49:07.664182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.664157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:49:07.688471 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.688444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:07.731782 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.731753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.731872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.731787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.731872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.731812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.731967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.731997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdtw\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.732029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.732052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.732110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.732217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.732188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdtw\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833605 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833671 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.833848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.833813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.834119 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.834098 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.835103 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.835076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:07.835808 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.835781 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.836033 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.836018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.842539 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.842513 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdtw\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.843051 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.843035 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token\") pod \"e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:07.959680 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:07.959653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:08.107844 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.107817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:49:08.109365 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:49:08.109338 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aa9a71_6974_49e0_956d_c9675abf6d63.slice/crio-ad5e6e20019700611c0c4c4d35ccf8694a186ddce4699363963fbcca84ac4703 WatchSource:0}: Error finding container ad5e6e20019700611c0c4c4d35ccf8694a186ddce4699363963fbcca84ac4703: Status 404 returned error can't find the container with id ad5e6e20019700611c0c4c4d35ccf8694a186ddce4699363963fbcca84ac4703 Jul 06 20:49:08.111364 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.111335 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:08.111441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.111398 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:08.111441 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.111425 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:08.376395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.376305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerStarted","Data":"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a"} Jul 06 20:49:08.376395 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.376358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerStarted","Data":"ad5e6e20019700611c0c4c4d35ccf8694a186ddce4699363963fbcca84ac4703"} Jul 06 20:49:08.402998 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.402950 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podStartSLOduration=1.402934567 podStartE2EDuration="1.402934567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:08.398423418 +0000 UTC m=+2064.889046970" watchObservedRunningTime="2026-07-06 20:49:08.402934567 +0000 UTC m=+2064.893558117" Jul 06 20:49:08.960327 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:08.960292 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:09.964947 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:09.964898 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:10.388773 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.388700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:10.389665 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.389645 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:10.469131 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.469101 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:10.560546 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.560516 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:49:10.561019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.561002 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="95db0404-88e4-452d-9bd6-668dbfab71c7" containerName="istio-proxy" Jul 06 20:49:10.563219 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.563199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.563590 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.563551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:10.565686 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.565670 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-7306a9-serving-cert\"" Jul 06 20:49:10.576951 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.576932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:49:10.661536 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.661382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.661536 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.661470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwqb\" (UniqueName: \"kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.661876 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.661572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.664259 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.664229 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:10.762257 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.762228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.762434 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.762306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.762434 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.762336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwqb\" (UniqueName: \"kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.762524 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:10.762453 2574 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-7306a9-serving-cert: secret "maas-api-e2e-shared-a-7306a9-serving-cert" not found Jul 06 20:49:10.762524 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:10.762514 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls podName:33053179-1ff0-4eef-8d86-33995eb228e4 nodeName:}" failed. No retries permitted until 2026-07-06 20:49:11.26249787 +0000 UTC m=+2067.753121399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls") pod "maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" (UID: "33053179-1ff0-4eef-8d86-33995eb228e4") : secret "maas-api-e2e-shared-a-7306a9-serving-cert" not found Jul 06 20:49:10.762814 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.762796 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.776524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.776494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwqb\" (UniqueName: \"kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:10.778138 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.778113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:10.874962 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:10.874942 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:11.003280 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.003204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:11.206469 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.206438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:11.266484 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.266417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:11.268770 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.268742 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") pod \"maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:11.313081 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.313052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:11.474391 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.474362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:11.827140 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.827111 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:49:11.827251 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:49:11.827191 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33053179_1ff0_4eef_8d86_33995eb228e4.slice/crio-663e73518f64c5ffa3b5f44000926d02d18d9f77af5af625225b1bea9dd16bb6 WatchSource:0}: Error finding container 663e73518f64c5ffa3b5f44000926d02d18d9f77af5af625225b1bea9dd16bb6: Status 404 returned error can't find the container with id 663e73518f64c5ffa3b5f44000926d02d18d9f77af5af625225b1bea9dd16bb6 Jul 06 20:49:11.968737 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:11.968712 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:49:12.074148 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074070 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074148 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074158 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv6pv\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074215 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074242 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074321 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074378 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074404 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs\") pod \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\" (UID: \"ab14378b-bce7-4f69-a072-97e5e6d2fad0\") " Jul 06 20:49:12.074612 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074581 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:12.075086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074754 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.075086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074772 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.075086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074776 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:12.075086 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.074798 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:49:12.075603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.075576 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:12.076505 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.076474 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:49:12.077406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.077354 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:49:12.077406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.077377 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv" (OuterVolumeSpecName: "kube-api-access-pv6pv") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "kube-api-access-pv6pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:49:12.082063 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.082039 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab14378b-bce7-4f69-a072-97e5e6d2fad0" (UID: "ab14378b-bce7-4f69-a072-97e5e6d2fad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:49:12.175404 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175383 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175404 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175404 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175414 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175421 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6pv\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-kube-api-access-pv6pv\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175432 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175440 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.175531 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.175448 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab14378b-bce7-4f69-a072-97e5e6d2fad0-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:49:12.401814 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.401782 2574 generic.go:356] "Generic (PLEG): container finished" podID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerID="d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f" exitCode=0 Jul 06 20:49:12.401937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.401834 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" Jul 06 20:49:12.401937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.401860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerDied","Data":"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f"} Jul 06 20:49:12.402029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.401940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt" event={"ID":"ab14378b-bce7-4f69-a072-97e5e6d2fad0","Type":"ContainerDied","Data":"9fa58f287269a4d451e95466a070ef65593c24994f76654a8fcb6a82f46bd56d"} Jul 06 20:49:12.402029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.401964 2574 scope.go:117] "RemoveContainer" containerID="d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f" Jul 06 20:49:12.403747 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.403725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" event={"ID":"33053179-1ff0-4eef-8d86-33995eb228e4","Type":"ContainerStarted","Data":"297167470add6a2107ba546906f90e6f3f2214d81ea83bdb2436ce6f382ee246"} Jul 06 20:49:12.403841 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.403769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" event={"ID":"33053179-1ff0-4eef-8d86-33995eb228e4","Type":"ContainerStarted","Data":"663e73518f64c5ffa3b5f44000926d02d18d9f77af5af625225b1bea9dd16bb6"} Jul 06 20:49:12.411064 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.411048 2574 scope.go:117] "RemoveContainer" containerID="8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484" Jul 06 20:49:12.419482 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.419462 2574 scope.go:117] "RemoveContainer" containerID="d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f" Jul 06 20:49:12.419722 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:12.419701 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f\": container with ID starting with d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f not found: ID does not exist" containerID="d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f" Jul 06 20:49:12.419766 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.419734 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f"} err="failed to get container status \"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f\": rpc error: code = NotFound desc = could not find container \"d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f\": container with ID starting with d1be54bcdf61234a1c50ac2c5d6db5082df05cb32197d222d16c649eb7504c8f not found: ID does not exist" Jul 06 20:49:12.419766 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.419757 2574 scope.go:117] "RemoveContainer" containerID="8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484" Jul 06 20:49:12.420040 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:49:12.420015 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484\": container with ID starting with 8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484 not found: ID does not exist" containerID="8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484" Jul 06 20:49:12.420100 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.420048 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484"} err="failed to get container status \"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484\": rpc error: code = NotFound desc = could not find container \"8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484\": container with ID starting with 8fcc60be6d4a82fd73de9c681b767ed8cf8219a1315655934e6b24c3dc329484 not found: ID does not exist" Jul 06 20:49:12.431848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.431806 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" podStartSLOduration=1.968989114 podStartE2EDuration="2.431781276s" totalImagesPullingTime="462.792162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:12.431627647 +0000 UTC m=+2068.922251198" watchObservedRunningTime="2026-07-06 20:49:12.431781276 +0000 UTC m=+2068.922404839" Jul 06 20:49:12.482213 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.482192 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:49:12.499403 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:12.499377 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b42d16-openshift-default-55445fc4f9-pl9gt"] Jul 06 20:49:13.413640 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:13.413571 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:14.072206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:14.072174 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" path="/var/lib/kubelet/pods/ab14378b-bce7-4f69-a072-97e5e6d2fad0/volumes" Jul 06 20:49:14.846645 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:14.846569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:15.188820 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.188791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:15.486605 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.486530 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:49:15.487189 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487163 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.487189 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487186 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.487349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487213 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.487349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487222 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.487349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487302 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.487349 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.487312 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab14378b-bce7-4f69-a072-97e5e6d2fad0" containerName="istio-proxy" Jul 06 20:49:15.490787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.490769 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.494486 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.494466 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d729fa-openshift-default-dockercfg-slnbn\"" Jul 06 20:49:15.504112 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.504091 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:49:15.514146 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.514125 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:15.604292 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604292 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604424 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604564 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf7h\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.604721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.604712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705401 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.705833 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705795 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.706024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf7h\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.706024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.705947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.706176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.706151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.706238 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.706172 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.706295 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.706234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.707919 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.707880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.708059 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.708044 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.713508 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.713491 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf7h\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.715597 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.715575 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token\") pod \"e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.801468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.801397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:15.862486 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.862452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:15.945303 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.945274 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:49:15.946352 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:49:15.946328 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb0d334_0a8b_4de6_be6d_ee50105c4f6d.slice/crio-b6dce3379f463f346aea57916461d9cd29b35b885f55c23aec8698ebd5a4fa8d WatchSource:0}: Error finding container b6dce3379f463f346aea57916461d9cd29b35b885f55c23aec8698ebd5a4fa8d: Status 404 returned error can't find the container with id b6dce3379f463f346aea57916461d9cd29b35b885f55c23aec8698ebd5a4fa8d Jul 06 20:49:15.948558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.948521 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:15.948676 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.948603 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:15.948676 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:15.948647 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:16.433169 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:16.433128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerStarted","Data":"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4"} Jul 06 20:49:16.433377 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:16.433195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerStarted","Data":"b6dce3379f463f346aea57916461d9cd29b35b885f55c23aec8698ebd5a4fa8d"} Jul 06 20:49:16.473553 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:16.473501 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podStartSLOduration=1.47348041 podStartE2EDuration="1.47348041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:16.471785563 +0000 UTC m=+2072.962409124" watchObservedRunningTime="2026-07-06 20:49:16.47348041 +0000 UTC m=+2072.964103960" Jul 06 20:49:16.802101 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:16.801954 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:17.808331 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:17.808302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:18.445308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.445271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:18.445526 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.445506 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:18.446431 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.446409 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:18.534366 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.534332 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:49:18.537124 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.537105 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.539197 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.539174 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-d729fa-serving-cert\"" Jul 06 20:49:18.549865 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.549842 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:49:18.563787 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.563760 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:18.632413 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.632377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.632589 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.632440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf5s5\" (UniqueName: \"kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.632589 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.632510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.669616 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.669565 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:18.733303 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.733221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.733459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.733309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.733459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.733352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf5s5\" (UniqueName: \"kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.733944 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.733897 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.735693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.735671 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.745829 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.745809 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf5s5\" (UniqueName: \"kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5\") pod \"maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:18.794583 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.794546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:18.849421 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:18.849398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:19.270134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:19.270103 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:49:19.271262 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:49:19.271237 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055d6ec3_8e11_4a42_9448_17a9ca660edd.slice/crio-6f0a5c22322d04d9df94e96553935f950f8532966ed05799ec5136a49a1a7a83 WatchSource:0}: Error finding container 6f0a5c22322d04d9df94e96553935f950f8532966ed05799ec5136a49a1a7a83: Status 404 returned error can't find the container with id 6f0a5c22322d04d9df94e96553935f950f8532966ed05799ec5136a49a1a7a83 Jul 06 20:49:19.329995 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:19.329965 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:19.423715 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:19.423693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:49:19.455054 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:19.455021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" event={"ID":"055d6ec3-8e11-4a42-9448-17a9ca660edd","Type":"ContainerStarted","Data":"6f0a5c22322d04d9df94e96553935f950f8532966ed05799ec5136a49a1a7a83"} Jul 06 20:49:19.463796 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:19.463769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:20.461602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:20.461566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" event={"ID":"055d6ec3-8e11-4a42-9448-17a9ca660edd","Type":"ContainerStarted","Data":"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732"} Jul 06 20:49:20.462025 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:20.461767 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:23.009602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.009567 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:23.488468 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.488441 2574 generic.go:356] "Generic (PLEG): container finished" podID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerID="1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a" exitCode=0 Jul 06 20:49:23.488640 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.488513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerDied","Data":"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a"} Jul 06 20:49:23.489013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.488985 2574 scope.go:117] "RemoveContainer" containerID="1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a" Jul 06 20:49:23.493655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.490753 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:23.493655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.490837 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:23.493655 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.490878 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:23.515660 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:23.515615 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" podStartSLOduration=5.140941474 podStartE2EDuration="5.515599512s" totalImagesPullingTime="374.658038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:49:20.486169623 +0000 UTC m=+2076.976793174" watchObservedRunningTime="2026-07-06 20:49:23.515599512 +0000 UTC m=+2080.006223063" Jul 06 20:49:24.502455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:24.502414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerStarted","Data":"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b"} Jul 06 20:49:24.960399 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:24.960365 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:24.961678 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:24.961649 2574 patch_prober.go:28] interesting pod/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Jul 06 20:49:24.961799 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:24.961701 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Jul 06 20:49:25.389208 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:25.389112 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:25.960419 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:25.960385 2574 patch_prober.go:28] interesting pod/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Jul 06 20:49:25.960816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:25.960433 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Jul 06 20:49:26.473962 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:26.473929 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:49:26.960912 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:26.960876 2574 patch_prober.go:28] interesting pod/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Jul 06 20:49:26.961406 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:26.960972 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Jul 06 20:49:27.960666 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:27.960631 2574 patch_prober.go:28] interesting pod/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Jul 06 20:49:27.960948 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:27.960691 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Jul 06 20:49:28.492473 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:28.492437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:28.592708 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:28.592670 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:28.964864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:28.964838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:28.965985 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:28.965964 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:49:33.292589 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.292552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:33.396779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.396750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:33.447257 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.447223 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 06 20:49:33.447542 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.447298 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 06 20:49:33.585864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.585776 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerID="8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4" exitCode=0 Jul 06 20:49:33.585864 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.585834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerDied","Data":"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4"} Jul 06 20:49:33.586281 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.586264 2574 scope.go:117] "RemoveContainer" containerID="8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4" Jul 06 20:49:33.587329 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.587300 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:33.587422 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.587399 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:33.587467 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:33.587440 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 20:49:34.597520 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:34.597476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerStarted","Data":"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e"} Jul 06 20:49:34.802075 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:34.802032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:34.802373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:34.802209 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 06 20:49:34.802456 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:34.802397 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 06 20:49:35.801946 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:35.801883 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 06 20:49:35.802524 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:35.802489 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 06 20:49:36.802029 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:36.801991 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 06 20:49:36.802486 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:36.802086 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 06 20:49:37.802223 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:37.802184 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 06 20:49:37.802606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:37.802279 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 06 20:49:38.805260 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:38.805232 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:38.805688 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:38.805669 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:38.806415 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:38.806399 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:49:41.251060 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:41.251025 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:41.350644 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:41.350603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:43.581809 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:43.581777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:49:44.136776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:44.136747 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:49:44.145308 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:44.145285 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:49:54.434269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:49:54.434239 2574 scope.go:117] "RemoveContainer" containerID="36adce538b89fed29f5649df3a33fe1a5a3d062696a3c798945637e92d066560" Jul 06 20:50:03.877058 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:50:03.877016 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:51:49.344336 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:51:49.344303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:51:50.469041 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:51:50.469010 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:51:54.473179 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:51:54.473141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:52:14.911347 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:52:14.911317 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:52:45.446405 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:52:45.446366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:52:45.518990 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:52:45.518960 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:52:50.046887 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:52:50.046851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:52:50.935577 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:52:50.935546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:53:00.369080 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:53:00.369041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:53:00.469769 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:53:00.469733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:53:02.454724 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:53:02.454691 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:53:11.884250 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:53:11.884207 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:53:40.274237 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:53:40.274157 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:30.633104 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:30.633071 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:31.535402 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:31.535372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:32.542329 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:32.542290 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:32.931697 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:32.931646 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:33.031825 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.031778 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:33.131650 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.131607 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:33.181861 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.181771 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:54:33.182299 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.182229 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" podUID="33053179-1ff0-4eef-8d86-33995eb228e4" containerName="maas-api" containerID="cri-o://297167470add6a2107ba546906f90e6f3f2214d81ea83bdb2436ce6f382ee246" gracePeriod=30 Jul 06 20:54:33.202211 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.202184 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:33.337989 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.337958 2574 generic.go:356] "Generic (PLEG): container finished" podID="33053179-1ff0-4eef-8d86-33995eb228e4" containerID="297167470add6a2107ba546906f90e6f3f2214d81ea83bdb2436ce6f382ee246" exitCode=0 Jul 06 20:54:33.338169 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.338028 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" event={"ID":"33053179-1ff0-4eef-8d86-33995eb228e4","Type":"ContainerDied","Data":"297167470add6a2107ba546906f90e6f3f2214d81ea83bdb2436ce6f382ee246"} Jul 06 20:54:33.440451 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.440399 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:54:33.472508 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.472485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwwqb\" (UniqueName: \"kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb\") pod \"33053179-1ff0-4eef-8d86-33995eb228e4\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " Jul 06 20:54:33.472647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.472535 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") pod \"33053179-1ff0-4eef-8d86-33995eb228e4\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " Jul 06 20:54:33.472647 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.472614 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca\") pod \"33053179-1ff0-4eef-8d86-33995eb228e4\" (UID: \"33053179-1ff0-4eef-8d86-33995eb228e4\") " Jul 06 20:54:33.472940 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.472897 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33053179-1ff0-4eef-8d86-33995eb228e4" (UID: "33053179-1ff0-4eef-8d86-33995eb228e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:54:33.474603 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.474582 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33053179-1ff0-4eef-8d86-33995eb228e4" (UID: "33053179-1ff0-4eef-8d86-33995eb228e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:54:33.474718 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.474618 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb" (OuterVolumeSpecName: "kube-api-access-rwwqb") pod "33053179-1ff0-4eef-8d86-33995eb228e4" (UID: "33053179-1ff0-4eef-8d86-33995eb228e4"). InnerVolumeSpecName "kube-api-access-rwwqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:54:33.573470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.573442 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33053179-1ff0-4eef-8d86-33995eb228e4-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:54:33.573470 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.573470 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwwqb\" (UniqueName: \"kubernetes.io/projected/33053179-1ff0-4eef-8d86-33995eb228e4-kube-api-access-rwwqb\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:54:33.573786 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:33.573486 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33053179-1ff0-4eef-8d86-33995eb228e4-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:54:34.345673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:34.345603 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" Jul 06 20:54:34.345673 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:34.345627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf" event={"ID":"33053179-1ff0-4eef-8d86-33995eb228e4","Type":"ContainerDied","Data":"663e73518f64c5ffa3b5f44000926d02d18d9f77af5af625225b1bea9dd16bb6"} Jul 06 20:54:34.345853 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:34.345685 2574 scope.go:117] "RemoveContainer" containerID="297167470add6a2107ba546906f90e6f3f2214d81ea83bdb2436ce6f382ee246" Jul 06 20:54:34.362648 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:34.362624 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:54:34.367784 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:34.367763 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-7306a9-85fc8789fc-z7mmf"] Jul 06 20:54:36.072024 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:36.071992 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33053179-1ff0-4eef-8d86-33995eb228e4" path="/var/lib/kubelet/pods/33053179-1ff0-4eef-8d86-33995eb228e4/volumes" Jul 06 20:54:38.018723 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:38.018692 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:39.557493 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:39.557463 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:42.100182 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:42.100141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:42.201674 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:42.201641 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:54:44.173394 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:44.173370 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:54:44.183539 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:54:44.183515 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6vmlz_1a4e3e5f-2493-490f-bbee-79748e604a3f/ovn-acl-logging/0.log" Jul 06 20:55:14.927450 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:14.927413 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:15.177642 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:15.177549 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:55:15.177884 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:15.177834 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" containerID="cri-o://ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b" gracePeriod=30 Jul 06 20:55:15.947942 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:15.947870 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:55:15.948779 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:15.948724 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" podUID="055d6ec3-8e11-4a42-9448-17a9ca660edd" containerName="maas-api" containerID="cri-o://5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732" gracePeriod=30 Jul 06 20:55:16.206179 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.206127 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:55:16.213826 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.213789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:16.345685 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.345652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls\") pod \"055d6ec3-8e11-4a42-9448-17a9ca660edd\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " Jul 06 20:55:16.345816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.345698 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf5s5\" (UniqueName: \"kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5\") pod \"055d6ec3-8e11-4a42-9448-17a9ca660edd\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " Jul 06 20:55:16.345816 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.345721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca\") pod \"055d6ec3-8e11-4a42-9448-17a9ca660edd\" (UID: \"055d6ec3-8e11-4a42-9448-17a9ca660edd\") " Jul 06 20:55:16.346102 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.346078 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "055d6ec3-8e11-4a42-9448-17a9ca660edd" (UID: "055d6ec3-8e11-4a42-9448-17a9ca660edd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:55:16.347753 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.347725 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "055d6ec3-8e11-4a42-9448-17a9ca660edd" (UID: "055d6ec3-8e11-4a42-9448-17a9ca660edd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:55:16.347847 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.347795 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5" (OuterVolumeSpecName: "kube-api-access-wf5s5") pod "055d6ec3-8e11-4a42-9448-17a9ca660edd" (UID: "055d6ec3-8e11-4a42-9448-17a9ca660edd"). InnerVolumeSpecName "kube-api-access-wf5s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:55:16.446671 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.446646 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/055d6ec3-8e11-4a42-9448-17a9ca660edd-maas-api-tls\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:16.446671 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.446668 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf5s5\" (UniqueName: \"kubernetes.io/projected/055d6ec3-8e11-4a42-9448-17a9ca660edd-kube-api-access-wf5s5\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:16.446800 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.446680 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/055d6ec3-8e11-4a42-9448-17a9ca660edd-openshift-service-ca\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:16.584834 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.584770 2574 generic.go:356] "Generic (PLEG): container finished" podID="055d6ec3-8e11-4a42-9448-17a9ca660edd" containerID="5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732" exitCode=0 Jul 06 20:55:16.584834 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.584807 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" Jul 06 20:55:16.584834 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.584811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" event={"ID":"055d6ec3-8e11-4a42-9448-17a9ca660edd","Type":"ContainerDied","Data":"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732"} Jul 06 20:55:16.585041 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.584863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4" event={"ID":"055d6ec3-8e11-4a42-9448-17a9ca660edd","Type":"ContainerDied","Data":"6f0a5c22322d04d9df94e96553935f950f8532966ed05799ec5136a49a1a7a83"} Jul 06 20:55:16.585041 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.584893 2574 scope.go:117] "RemoveContainer" containerID="5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732" Jul 06 20:55:16.598073 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.598051 2574 scope.go:117] "RemoveContainer" containerID="5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732" Jul 06 20:55:16.598321 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:55:16.598300 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732\": container with ID starting with 5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732 not found: ID does not exist" containerID="5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732" Jul 06 20:55:16.598389 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.598334 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732"} err="failed to get container status \"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732\": rpc error: code = NotFound desc = could not find container \"5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732\": container with ID starting with 5d1989f20a0d8346edc870abcd3f512ee267ecf59e1988912bb8b2fc842e9732 not found: ID does not exist" Jul 06 20:55:16.608776 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.608752 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:55:16.612958 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:16.612938 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-d729fa-9875c76bf-wbrv4"] Jul 06 20:55:18.072710 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:18.072678 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="055d6ec3-8e11-4a42-9448-17a9ca660edd" path="/var/lib/kubelet/pods/055d6ec3-8e11-4a42-9448-17a9ca660edd/volumes" Jul 06 20:55:20.428525 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.428498 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:55:20.485139 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485139 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485313 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485313 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwdtw\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485313 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485237 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485313 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485313 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485333 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485355 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket\") pod \"b3aa9a71-6974-49e0-956d-c9675abf6d63\" (UID: \"b3aa9a71-6974-49e0-956d-c9675abf6d63\") " Jul 06 20:55:20.485565 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485495 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:20.485744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485577 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:20.485744 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485699 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:20.485871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485820 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.485871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485837 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.485871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485851 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.485871 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.485864 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:55:20.486118 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.486015 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:20.487444 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.487424 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:20.487552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.487454 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw" (OuterVolumeSpecName: "kube-api-access-rwdtw") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "kube-api-access-rwdtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:55:20.487552 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.487516 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:55:20.487636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.487599 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3aa9a71-6974-49e0-956d-c9675abf6d63" (UID: "b3aa9a71-6974-49e0-956d-c9675abf6d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:55:20.586978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.586953 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3aa9a71-6974-49e0-956d-c9675abf6d63-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.586978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.586976 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.586978 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.586984 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.587155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.586992 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdtw\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-kube-api-access-rwdtw\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.587155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.587002 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.587155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.587009 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3aa9a71-6974-49e0-956d-c9675abf6d63-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:20.610402 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.610372 2574 generic.go:356] "Generic (PLEG): container finished" podID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerID="ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b" exitCode=0 Jul 06 20:55:20.610512 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.610422 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" Jul 06 20:55:20.610512 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.610460 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerDied","Data":"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b"} Jul 06 20:55:20.610653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.610541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc" event={"ID":"b3aa9a71-6974-49e0-956d-c9675abf6d63","Type":"ContainerDied","Data":"ad5e6e20019700611c0c4c4d35ccf8694a186ddce4699363963fbcca84ac4703"} Jul 06 20:55:20.610653 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.610567 2574 scope.go:117] "RemoveContainer" containerID="ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b" Jul 06 20:55:20.620815 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.620778 2574 scope.go:117] "RemoveContainer" containerID="1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a" Jul 06 20:55:20.628661 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.628644 2574 scope.go:117] "RemoveContainer" containerID="ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b" Jul 06 20:55:20.628891 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:55:20.628871 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b\": container with ID starting with ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b not found: ID does not exist" containerID="ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b" Jul 06 20:55:20.628986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.628915 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b"} err="failed to get container status \"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b\": rpc error: code = NotFound desc = could not find container \"ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b\": container with ID starting with ae0f0512b7ac3dc38606ded8083d6e4e55b68687d41f8990812c085f6c971f5b not found: ID does not exist" Jul 06 20:55:20.628986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.628939 2574 scope.go:117] "RemoveContainer" containerID="1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a" Jul 06 20:55:20.629184 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:55:20.629165 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a\": container with ID starting with 1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a not found: ID does not exist" containerID="1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a" Jul 06 20:55:20.629229 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.629190 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a"} err="failed to get container status \"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a\": rpc error: code = NotFound desc = could not find container \"1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a\": container with ID starting with 1e6374baa297c1177d91ff6aa6ff84210f66ab35d379208b2be5e3eca3de9c0a not found: ID does not exist" Jul 06 20:55:20.632935 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.632898 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:55:20.636714 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:20.636695 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7306a9-openshift-default-ddc5875f9-ck9pc"] Jul 06 20:55:21.992220 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:21.992190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:22.073007 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.072977 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" path="/var/lib/kubelet/pods/b3aa9a71-6974-49e0-956d-c9675abf6d63/volumes" Jul 06 20:55:22.095212 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.095183 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:22.225305 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.225237 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:55:22.225527 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.225435 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" containerID="cri-o://b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e" gracePeriod=30 Jul 06 20:55:22.247973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.247876 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:22.468449 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.468420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:22.873322 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:22.873294 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbbfdc888-gvdqk_91f0fc03-9347-4fff-8753-6d65e0746a0c/authorino/0.log" Jul 06 20:55:23.807407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:23.807379 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 20:55:23.807775 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:23.807440 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 20:55:27.435927 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.435876 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6fnqj_35369594-ea72-4a81-a41c-7298f7f73965/manager/0.log" Jul 06 20:55:27.480965 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.480943 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:55:27.546729 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546654 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgf7h\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546729 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546704 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546753 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546798 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546822 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546851 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.546977 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.546888 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.547380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.547380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547038 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo\") pod \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\" (UID: \"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d\") " Jul 06 20:55:27.547380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547107 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:27.547380 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547185 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:55:27.547670 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547431 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.547670 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547455 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istiod-ca-cert\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.547808 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547672 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:27.547966 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547940 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:27.548043 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.547965 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:27.549378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.549350 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:55:27.549532 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.549513 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:55:27.549602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.549534 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 20:55:27.549773 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.549757 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h" (OuterVolumeSpecName: "kube-api-access-xgf7h") pod "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" (UID: "0bb0d334-0a8b-4de6-be6d-ee50105c4f6d"). InnerVolumeSpecName "kube-api-access-xgf7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:55:27.551234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.551216 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wpkdn_4b7e81c5-75bc-47c8-b95a-7b3d13aedd24/manager/0.log" Jul 06 20:55:27.648048 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648024 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-envoy\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648048 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648044 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-podinfo\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648055 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgf7h\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-kube-api-access-xgf7h\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648063 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-token\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648072 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-workload-certs\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648079 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-istio-data\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.648173 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.648086 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d-credential-socket\") on node \"ip-10-0-130-104.ec2.internal\" DevicePath \"\"" Jul 06 20:55:27.656870 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.656847 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerID="b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e" exitCode=0 Jul 06 20:55:27.656986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.656891 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" Jul 06 20:55:27.656986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.656948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerDied","Data":"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e"} Jul 06 20:55:27.657077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.657010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr" event={"ID":"0bb0d334-0a8b-4de6-be6d-ee50105c4f6d","Type":"ContainerDied","Data":"b6dce3379f463f346aea57916461d9cd29b35b885f55c23aec8698ebd5a4fa8d"} Jul 06 20:55:27.657077 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.657041 2574 scope.go:117] "RemoveContainer" containerID="b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e" Jul 06 20:55:27.667539 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.667524 2574 scope.go:117] "RemoveContainer" containerID="8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4" Jul 06 20:55:27.671952 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.671899 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-688c79454f-bqw54_61ef62ac-1d3d-4028-a95c-4c219060a3c9/maas-api/0.log" Jul 06 20:55:27.676712 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.676692 2574 scope.go:117] "RemoveContainer" containerID="b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e" Jul 06 20:55:27.676972 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:55:27.676951 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e\": container with ID starting with b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e not found: ID does not exist" containerID="b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e" Jul 06 20:55:27.677032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.676982 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e"} err="failed to get container status \"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e\": rpc error: code = NotFound desc = could not find container \"b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e\": container with ID starting with b51a16bef1742f233461ba50e9fec74ad0484e7d6ea2216bd6450132a985c19e not found: ID does not exist" Jul 06 20:55:27.677032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.677000 2574 scope.go:117] "RemoveContainer" containerID="8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4" Jul 06 20:55:27.677246 ip-10-0-130-104 kubenswrapper[2574]: E0706 20:55:27.677226 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4\": container with ID starting with 8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4 not found: ID does not exist" containerID="8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4" Jul 06 20:55:27.677304 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.677257 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4"} err="failed to get container status \"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4\": rpc error: code = NotFound desc = could not find container \"8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4\": container with ID starting with 8a757495d545ffb70691a42a1a4ff4bb560b56bb70711a039d42b2ed17dcacb4 not found: ID does not exist" Jul 06 20:55:27.681269 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.681223 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:55:27.684354 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.684335 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d729fa-openshift-default-59c5547d47-mgwxr"] Jul 06 20:55:27.781223 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:27.781203 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722830-2tjsd_6314cea7-9e22-41e0-bcc9-640df1c4cc5f/cleanup/0.log" Jul 06 20:55:28.019693 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.019662 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4b87b-vnphq_35db64d2-85c0-4c6f-99b7-69608b972e3b/manager/0.log" Jul 06 20:55:28.072872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.072839 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" path="/var/lib/kubelet/pods/0bb0d334-0a8b-4de6-be6d-ee50105c4f6d/volumes" Jul 06 20:55:28.136127 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.136103 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sl4zh_19494732-31c5-43ef-a501-d15af99b6892/server/0.log" Jul 06 20:55:28.251154 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.251129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rmsfb_85d53d1d-83cb-4a48-a038-0848e25232f3/manager/0.log" Jul 06 20:55:28.492674 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.492652 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf_2b1a752a-c485-4755-9cfd-9a4ec86691f8/copy-manifests/0.log" Jul 06 20:55:28.501868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.501839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf_2b1a752a-c485-4755-9cfd-9a4ec86691f8/manager/0.log" Jul 06 20:55:28.752972 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:28.752890 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-22d2g_f5397d8c-4e76-4503-be19-76022375845c/postgres/0.log" Jul 06 20:55:29.558592 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.558564 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/pull/0.log" Jul 06 20:55:29.564751 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.564713 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/extract/0.log" Jul 06 20:55:29.571108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.571082 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/util/0.log" Jul 06 20:55:29.682223 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.682200 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/util/0.log" Jul 06 20:55:29.689104 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.689081 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/pull/0.log" Jul 06 20:55:29.695828 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.695812 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/extract/0.log" Jul 06 20:55:29.809530 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.809471 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/util/0.log" Jul 06 20:55:29.815949 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.815934 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/pull/0.log" Jul 06 20:55:29.822129 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.822109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/extract/0.log" Jul 06 20:55:29.929967 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.929943 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/util/0.log" Jul 06 20:55:29.936459 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.936442 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/pull/0.log" Jul 06 20:55:29.942574 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:29.942553 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/extract/0.log" Jul 06 20:55:30.054019 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:30.053984 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbbfdc888-gvdqk_91f0fc03-9347-4fff-8753-6d65e0746a0c/authorino/0.log" Jul 06 20:55:30.168621 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:30.168590 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-lk47z_5512e65e-c821-4757-aae1-bfe75e050e71/manager/0.log" Jul 06 20:55:30.396384 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:30.396355 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jp898_c917b062-7153-4753-901e-9ec634d1e77f/kuadrant-console-plugin/0.log" Jul 06 20:55:30.510952 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:30.510878 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vfrmm_b3c43d88-0d0a-4647-870f-7550e4e7f9c5/registry-server/0.log" Jul 06 20:55:30.631553 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:30.631529 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2hzfz_4469b32d-138a-43fb-abc2-4b60b8e75adc/manager/0.log" Jul 06 20:55:31.225217 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:31.225177 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k5z2p_2ca16ad5-466a-4cb5-8d3b-df72530c6e57/istio-proxy/0.log" Jul 06 20:55:31.582363 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:31.582293 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f9c87b66f-5kn9d_d3c53cfd-f962-4ef1-8de9-ba591d9e49e1/kube-auth-proxy/0.log" Jul 06 20:55:31.702543 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:31.702520 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/1.log" Jul 06 20:55:32.384941 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:32.384886 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-w65pb_63b1ec5a-b67b-4571-b65b-ce9e30d92e36/main/0.log" Jul 06 20:55:32.722964 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:32.722938 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbb56ps_f898e230-9f00-49e4-b750-a66ffb0297a1/main/0.log" Jul 06 20:55:37.294629 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.294598 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hvwvn/must-gather-87wbn"] Jul 06 20:55:37.295032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295019 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.295032 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295034 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295052 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295058 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295065 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33053179-1ff0-4eef-8d86-33995eb228e4" containerName="maas-api" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295071 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="33053179-1ff0-4eef-8d86-33995eb228e4" containerName="maas-api" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295079 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295085 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295092 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055d6ec3-8e11-4a42-9448-17a9ca660edd" containerName="maas-api" Jul 06 20:55:37.295108 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295098 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="055d6ec3-8e11-4a42-9448-17a9ca660edd" containerName="maas-api" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295156 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295163 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="055d6ec3-8e11-4a42-9448-17a9ca660edd" containerName="maas-api" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295170 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295191 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="33053179-1ff0-4eef-8d86-33995eb228e4" containerName="maas-api" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295198 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295204 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb0d334-0a8b-4de6-be6d-ee50105c4f6d" containerName="istio-proxy" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295255 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.295346 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.295260 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3aa9a71-6974-49e0-956d-c9675abf6d63" containerName="istio-proxy" Jul 06 20:55:37.300014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.299994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.302575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.302557 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hvwvn\"/\"openshift-service-ca.crt\"" Jul 06 20:55:37.302575 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.302568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hvwvn\"/\"default-dockercfg-dk4dk\"" Jul 06 20:55:37.302719 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.302617 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hvwvn\"/\"kube-root-ca.crt\"" Jul 06 20:55:37.307604 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.307580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hvwvn/must-gather-87wbn"] Jul 06 20:55:37.309134 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.309106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:37.328292 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.328270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhl7\" (UniqueName: \"kubernetes.io/projected/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-kube-api-access-vqhl7\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.328376 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.328308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-must-gather-output\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.408848 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.408829 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:37.428720 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.428701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhl7\" (UniqueName: \"kubernetes.io/projected/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-kube-api-access-vqhl7\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.428803 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.428739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-must-gather-output\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.429022 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.429004 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-must-gather-output\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.441396 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.441371 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhl7\" (UniqueName: \"kubernetes.io/projected/d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4-kube-api-access-vqhl7\") pod \"must-gather-87wbn\" (UID: \"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4\") " pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.611558 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.611503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hvwvn/must-gather-87wbn" Jul 06 20:55:37.745716 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.745685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hvwvn/must-gather-87wbn"] Jul 06 20:55:37.746349 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:55:37.746323 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd41c1c6d_69bb_4783_84f9_d2a14dfcdbd4.slice/crio-c129166b86aa86d8baea73bc71958ef1a30222977b58157f899ba73976d8e83a WatchSource:0}: Error finding container c129166b86aa86d8baea73bc71958ef1a30222977b58157f899ba73976d8e83a: Status 404 returned error can't find the container with id c129166b86aa86d8baea73bc71958ef1a30222977b58157f899ba73976d8e83a Jul 06 20:55:37.748123 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:37.748104 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:55:38.719541 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:38.719509 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hvwvn/must-gather-87wbn" event={"ID":"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4","Type":"ContainerStarted","Data":"c129166b86aa86d8baea73bc71958ef1a30222977b58157f899ba73976d8e83a"} Jul 06 20:55:39.730657 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:39.730612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hvwvn/must-gather-87wbn" event={"ID":"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4","Type":"ContainerStarted","Data":"5d488390232879f3db132c026b04bfc62d1c755ba1d6fd3ea75e14ad8e43e3f9"} Jul 06 20:55:39.731163 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:39.730670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hvwvn/must-gather-87wbn" event={"ID":"d41c1c6d-69bb-4783-84f9-d2a14dfcdbd4","Type":"ContainerStarted","Data":"a9a4193a0feb85d649a10b6013e43f4b26a8ed6ae610914d1837b060cda27f9e"} Jul 06 20:55:39.747986 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:39.747936 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hvwvn/must-gather-87wbn" podStartSLOduration=1.7194704459999999 podStartE2EDuration="2.747918381s" totalImagesPullingTime="1.028447935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:55:39.745640736 +0000 UTC m=+2456.236264300" watchObservedRunningTime="2026-07-06 20:55:39.747918381 +0000 UTC m=+2456.238541933" Jul 06 20:55:40.337960 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:40.337925 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mpq6z_5fd1d381-faee-4d65-9d12-05c0a91abc96/global-pull-secret-syncer/0.log" Jul 06 20:55:40.468868 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:40.468836 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" Jul 06 20:55:40.562790 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:40.562764 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-104.ec2.internal_5457eba2732788e8cf4686bd84b1d393/haproxy/0.log" Jul 06 20:55:44.280533 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.280508 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/extract/0.log" Jul 06 20:55:44.316013 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.315974 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/util/0.log" Jul 06 20:55:44.342186 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.342057 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594kkzd_2d2ea925-982a-4417-a43a-255d64b2e864/pull/0.log" Jul 06 20:55:44.376753 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.376721 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/extract/0.log" Jul 06 20:55:44.399378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.399341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/util/0.log" Jul 06 20:55:44.431721 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.431663 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0pqgrx_2b4d62be-a38d-48c4-813e-eb6de01bff2a/pull/0.log" Jul 06 20:55:44.462513 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.462460 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/extract/0.log" Jul 06 20:55:44.486294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.486262 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/util/0.log" Jul 06 20:55:44.510021 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.509994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73m96lv_dd88a946-654f-4e5d-a400-18d4b3a3ac3b/pull/0.log" Jul 06 20:55:44.545423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.545335 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/extract/0.log" Jul 06 20:55:44.569992 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.569962 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/util/0.log" Jul 06 20:55:44.589840 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:44.589769 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1kg5mk_8942ef1e-616a-4fc1-949c-36de262b1c6f/pull/0.log" Jul 06 20:55:45.005923 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:45.005052 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cbbfdc888-gvdqk_91f0fc03-9347-4fff-8753-6d65e0746a0c/authorino/0.log" Jul 06 20:55:45.038183 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:45.038088 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-lk47z_5512e65e-c821-4757-aae1-bfe75e050e71/manager/0.log" Jul 06 20:55:45.118937 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:45.118887 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jp898_c917b062-7153-4753-901e-9ec634d1e77f/kuadrant-console-plugin/0.log" Jul 06 20:55:45.166656 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:45.166611 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vfrmm_b3c43d88-0d0a-4647-870f-7550e4e7f9c5/registry-server/0.log" Jul 06 20:55:45.364973 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:45.362559 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2hzfz_4469b32d-138a-43fb-abc2-4b60b8e75adc/manager/0.log" Jul 06 20:55:47.189373 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.189337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-flbkr_400fd57c-a0e7-4bf1-a060-0ba19f3103b1/kube-state-metrics/0.log" Jul 06 20:55:47.219209 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.219177 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-flbkr_400fd57c-a0e7-4bf1-a060-0ba19f3103b1/kube-rbac-proxy-main/0.log" Jul 06 20:55:47.251925 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.251486 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-flbkr_400fd57c-a0e7-4bf1-a060-0ba19f3103b1/kube-rbac-proxy-self/0.log" Jul 06 20:55:47.303701 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.302972 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f5db85cbc-4gzqt_20896075-806c-45f8-8c5e-2454448b8adc/metrics-server/0.log" Jul 06 20:55:47.570659 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.570583 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqg8n_2387edeb-79d8-49fd-b9fd-b6acab436a83/node-exporter/0.log" Jul 06 20:55:47.599465 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.599436 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqg8n_2387edeb-79d8-49fd-b9fd-b6acab436a83/kube-rbac-proxy/0.log" Jul 06 20:55:47.625176 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.625153 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqg8n_2387edeb-79d8-49fd-b9fd-b6acab436a83/init-textfile/0.log" Jul 06 20:55:47.661602 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.661578 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z22m2_75782c50-01a4-43b8-b4af-8afff91c5ead/kube-rbac-proxy-main/0.log" Jul 06 20:55:47.693049 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.693022 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z22m2_75782c50-01a4-43b8-b4af-8afff91c5ead/kube-rbac-proxy-self/0.log" Jul 06 20:55:47.720432 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:47.720404 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-z22m2_75782c50-01a4-43b8-b4af-8afff91c5ead/openshift-state-metrics/0.log" Jul 06 20:55:50.179606 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.179563 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:55:50.317401 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.317352 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j"] Jul 06 20:55:50.325971 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.325942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.334337 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.334310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j"] Jul 06 20:55:50.490372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.490285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-proc\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.490372 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.490329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-lib-modules\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.490566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.490405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82jp\" (UniqueName: \"kubernetes.io/projected/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-kube-api-access-c82jp\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.490566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.490501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-podres\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.490566 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.490554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-sys\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.591786 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.591740 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-proc\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.591812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-lib-modules\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.591875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82jp\" (UniqueName: \"kubernetes.io/projected/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-kube-api-access-c82jp\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.591927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-podres\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592014 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.591973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-sys\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.592110 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-sys\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592234 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.592176 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-proc\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592351 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.592290 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-lib-modules\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.592696 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.592673 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-podres\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.602059 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.602036 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82jp\" (UniqueName: \"kubernetes.io/projected/b1ce39f7-75cc-4a49-8b93-a4c13bceadd9-kube-api-access-c82jp\") pod \"perf-node-gather-daemonset-bsl4j\" (UID: \"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9\") " pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.643141 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.643106 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:50.856605 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.856290 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j"] Jul 06 20:55:50.860282 ip-10-0-130-104 kubenswrapper[2574]: W0706 20:55:50.860246 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1ce39f7_75cc_4a49_8b93_a4c13bceadd9.slice/crio-bbe6d2510f02a6d94b0b9e06fc105b9c718cb78771c1636b53370b5ffd4e2c2b WatchSource:0}: Error finding container bbe6d2510f02a6d94b0b9e06fc105b9c718cb78771c1636b53370b5ffd4e2c2b: Status 404 returned error can't find the container with id bbe6d2510f02a6d94b0b9e06fc105b9c718cb78771c1636b53370b5ffd4e2c2b Jul 06 20:55:50.891166 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.891130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" event={"ID":"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9","Type":"ContainerStarted","Data":"bbe6d2510f02a6d94b0b9e06fc105b9c718cb78771c1636b53370b5ffd4e2c2b"} Jul 06 20:55:50.896959 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:50.896941 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d96c4dbcd-9rjdj_069ec0b6-c77f-407e-800a-5722cdaa6fd3/console/0.log" Jul 06 20:55:51.901378 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:51.901304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" event={"ID":"b1ce39f7-75cc-4a49-8b93-a4c13bceadd9","Type":"ContainerStarted","Data":"b49d9bb97062354fd8a8666b5f11ea710d02a1de57c33e68ad29be179176ffe5"} Jul 06 20:55:51.901975 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:51.901897 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:51.919732 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:51.919671 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" podStartSLOduration=1.919652334 podStartE2EDuration="1.919652334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:55:51.917759811 +0000 UTC m=+2468.408383363" watchObservedRunningTime="2026-07-06 20:55:51.919652334 +0000 UTC m=+2468.410275884" Jul 06 20:55:52.562265 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:52.562223 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pxw7m_13edad41-5e3f-4825-a0ce-9004a505e33f/dns/0.log" Jul 06 20:55:52.582727 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:52.582704 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pxw7m_13edad41-5e3f-4825-a0ce-9004a505e33f/kube-rbac-proxy/0.log" Jul 06 20:55:52.730066 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:52.730027 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b88jl_edfb46ee-7922-4f25-8411-0f5c8c6554e0/dns-node-resolver/0.log" Jul 06 20:55:53.289785 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:53.289753 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c9b96759f-dmn9j_ce5bf723-805b-415f-b2c1-ce704b1b1b65/registry/0.log" Jul 06 20:55:53.313507 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:53.313481 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4djqt_e59897e4-965e-4df8-899d-70fcfe6d1ed8/node-ca/0.log" Jul 06 20:55:54.300500 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:54.300461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k5z2p_2ca16ad5-466a-4cb5-8d3b-df72530c6e57/istio-proxy/0.log" Jul 06 20:55:54.560155 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:54.560095 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f9c87b66f-5kn9d_d3c53cfd-f962-4ef1-8de9-ba591d9e49e1/kube-auth-proxy/0.log" Jul 06 20:55:54.591423 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:54.591400 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/0.log" Jul 06 20:55:54.625272 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:54.625244 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/1.log" Jul 06 20:55:55.430159 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:55.430133 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbhmr_e0d27bad-d36b-409c-afdd-8bc6ac3a8e91/serve-healthcheck-canary/0.log" Jul 06 20:55:56.005324 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:56.005296 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glwkf_c8be60e6-c948-4081-a1ea-e1fe0012e849/kube-rbac-proxy/0.log" Jul 06 20:55:56.028873 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:56.028842 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glwkf_c8be60e6-c948-4081-a1ea-e1fe0012e849/exporter/0.log" Jul 06 20:55:56.052188 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:56.052161 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glwkf_c8be60e6-c948-4081-a1ea-e1fe0012e849/extractor/0.log" Jul 06 20:55:57.918317 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:57.918284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hvwvn/perf-node-gather-daemonset-bsl4j" Jul 06 20:55:58.240206 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.240128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6fnqj_35369594-ea72-4a81-a41c-7298f7f73965/manager/0.log" Jul 06 20:55:58.416031 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.416006 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wpkdn_4b7e81c5-75bc-47c8-b95a-7b3d13aedd24/manager/0.log" Jul 06 20:55:58.460455 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.460413 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-688c79454f-bqw54_61ef62ac-1d3d-4028-a95c-4c219060a3c9/maas-api/0.log" Jul 06 20:55:58.484556 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.484530 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722830-2tjsd_6314cea7-9e22-41e0-bcc9-640df1c4cc5f/cleanup/0.log" Jul 06 20:55:58.646636 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.646558 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d79b4b87b-vnphq_35db64d2-85c0-4c6f-99b7-69608b972e3b/manager/0.log" Jul 06 20:55:58.670015 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.669986 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sl4zh_19494732-31c5-43ef-a501-d15af99b6892/server/0.log" Jul 06 20:55:58.745294 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.745261 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rmsfb_85d53d1d-83cb-4a48-a038-0848e25232f3/manager/0.log" Jul 06 20:55:58.891243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.891215 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf_2b1a752a-c485-4755-9cfd-9a4ec86691f8/manager/0.log" Jul 06 20:55:58.913872 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.913852 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9b6b7b7bb-hzwpf_2b1a752a-c485-4755-9cfd-9a4ec86691f8/copy-manifests/0.log" Jul 06 20:55:58.994153 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:55:58.994125 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-22d2g_f5397d8c-4e76-4503-be19-76022375845c/postgres/0.log" Jul 06 20:56:00.561826 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:00.561792 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k5z2p_2ca16ad5-466a-4cb5-8d3b-df72530c6e57/istio-proxy/0.log" Jul 06 20:56:00.790005 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:00.789966 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f9c87b66f-5kn9d_d3c53cfd-f962-4ef1-8de9-ba591d9e49e1/kube-auth-proxy/0.log" Jul 06 20:56:00.811180 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:00.811152 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/0.log" Jul 06 20:56:00.848876 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:00.848788 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/1.log" Jul 06 20:56:02.083826 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:02.083795 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k5z2p_2ca16ad5-466a-4cb5-8d3b-df72530c6e57/istio-proxy/0.log" Jul 06 20:56:02.302823 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:02.302797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f9c87b66f-5kn9d_d3c53cfd-f962-4ef1-8de9-ba591d9e49e1/kube-auth-proxy/0.log" Jul 06 20:56:02.325523 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:02.325488 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/0.log" Jul 06 20:56:02.360874 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:02.360844 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2qk7c_d94c2af7-b222-4c58-97b0-528de511c863/istio-proxy/1.log" Jul 06 20:56:03.289407 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:03.289366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5clp9_7fb82de6-3f26-4671-89d1-4b13446422bf/konnectivity-agent/0.log" limit=1024 Jul 06 20:56:07.004140 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.004109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/kube-multus-additional-cni-plugins/0.log" Jul 06 20:56:07.027629 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.027603 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/egress-router-binary-copy/0.log" Jul 06 20:56:07.051462 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.051421 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/cni-plugins/0.log" Jul 06 20:56:07.073221 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.073198 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/bond-cni-plugin/0.log" Jul 06 20:56:07.097243 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.097227 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/routeoverride-cni/0.log" Jul 06 20:56:07.120894 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.120875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/whereabouts-cni-bincopy/0.log" Jul 06 20:56:07.141671 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.141649 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rm2rx_9b7ec425-fa4f-4648-8e08-561cb0cb7984/whereabouts-cni/0.log" Jul 06 20:56:07.354481 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.354405 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cmtzl_efc56e0e-e63c-4976-b894-0775506ea70b/kube-multus/0.log" Jul 06 20:56:07.464679 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.464656 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d9crx_de5a9039-dd79-406d-96c8-0e10737ecf6d/network-metrics-daemon/0.log" Jul 06 20:56:07.486597 ip-10-0-130-104 kubenswrapper[2574]: I0706 20:56:07.486574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d9crx_de5a9039-dd79-406d-96c8-0e10737ecf6d/kube-rbac-proxy/0.log"